2022-03-08T22:50:24.124 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:50:24.126 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} --archive /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727292 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727292/orig.config.yaml 2022-03-08T22:50:24.162 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:50:24.223 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727292 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} email: null first_in_suite: false job_id: '6727292' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi117.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDarK5jtVyD6PM4uJVfooU/jpki+1ibjSz9ahTYRm96vMlFXrJgkYO+cJ64j1YXuNdbGo8KAFlp53JrKAq6HuYaX3tlCcTbVlYcs1J+zmTTJN02YWsagFBCyxbfamXP4LC3ReIsVZr4czCStFHfkwIPnLcOFsLIBRhtSYMDyyDBICv/T/ba1Xqy3VIKU4MH7TJnN5a2MfHZKjV35Nx7q5288L2Mi3A2tSAPAjbFP6rjY0tmT63v7DNEc69lM5cb1MYmTalz6BuOHnBBQgMyLK9g2Ak17MFWb5DxVt7EmL67ToSqmeSbt/wUJ1zkxTmysnwd4yPyakOm+2aZQuLog5IswZtBCh5aUsVX+e7tRc/mP13NIu3ugFa5nZ2t7Wd/rNf5ZVWIAkJ8bMSGcocRvPc5yE0riqvXJ+z/bsntUhUVzW/HEdfpiiBQCWHVoA+SM3zXGpenLK2E3H5s34ZhPt9AfdzNiyCX0i3SHiA4SbUWDMtwRvtXRlnFbKvYBdVSC18= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:38:39 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:50:24.224 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-08T22:50:24.225 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-08T22:50:24.225 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:50:24.227 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:50:24.247 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-08T22:50:24.248 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:50:24.248 INFO:teuthology.packaging:ref: None 2022-03-08T22:50:24.249 INFO:teuthology.packaging:tag: None 2022-03-08T22:50:24.249 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:50:24.249 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T22:50:24.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:50:24.416 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T22:50:24.416 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:50:24.426 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:50:24.426 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:50:24.477 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:50:24.490 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:50:24.511 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:50:24.535 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727292', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:44:59.457614', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILOK1k1b/xeGH8A0SdAI1SyHAb1g1Tcs5L5q/5DzgQtc'} 2022-03-08T22:50:24.535 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:50:24.548 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:50:24.548 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:50:24.608 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2dc2ca69d8>, signals=[15]) 2022-03-08T22:50:24.609 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:50:24.619 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:50:24.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2022-03-08T22:50:24.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:50:24.674 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:50:24.684 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2022-03-08T22:50:24.801 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2022-03-08T22:50:24.801 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2022-03-08T22:50:24.808 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="Ubuntu" 2022-03-08T22:50:24.808 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:50:24.809 INFO:teuthology.orchestra.run.smithi117.stdout:ID=ubuntu 2022-03-08T22:50:24.809 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE=debian 2022-03-08T22:50:24.809 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:50:24.809 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="20.04" 2022-03-08T22:50:24.809 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:50:24.810 INFO:teuthology.orchestra.run.smithi117.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:50:24.810 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:50:24.810 INFO:teuthology.orchestra.run.smithi117.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:50:24.810 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_CODENAME=focal 2022-03-08T22:50:24.811 INFO:teuthology.orchestra.run.smithi117.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:50:24.812 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2022-03-08T22:50:24.867 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:50:24.880 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:50:24.939 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:50:24.940 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:50:24.947 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:50:24.969 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:50:24.969 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:50:24.997 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:50:25.029 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:50:25.049 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:50:25.050 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:50:25.050 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-08T22:50:25.050 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:50:25.051 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2022-03-08T22:50:25.058 INFO:teuthology.orchestra.run.smithi117.stdout:5.4.0-97-generic 2022-03-08T22:50:25.058 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.4.0-97-generic 2022-03-08T22:50:25.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2022-03-08T22:50:25.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2022-03-08T22:50:25.274 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:50:25.389 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:50:25.472 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:50:25.608 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:50:25.722 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:50:25.735 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:50:25.735 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:50:25.736 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:50:25.736 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:50:25.742 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:50:25.743 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:50:25.743 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:50:25.750 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:50:25.756 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:50:25.758 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:50:25.758 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:50:25.758 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:50:25.776 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:50:25.777 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:50:25.809 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:50:25.927 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:50:26.282 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:50:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:50:26.317 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:50:26.318 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:50:26.364 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:50:26.365 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:50:26.368 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:50:26.368 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:50:26.403 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:50:26.422 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:50:26.427 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:50:26.451 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:50:26.451 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:50:26.452 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:50:26.452 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:50:26.453 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:50:26.472 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:50:26.554 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:50:26.555 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:50:26.555 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:50:26.556 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:50:26.556 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:50:29.061 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 10.2 MB in 2s (6,274 kB/s) 2022-03-08T22:50:30.300 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:50:30.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:50:30.393 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:50:30.565 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:50:30.566 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:50:30.697 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2022-03-08T22:50:30.698 INFO:teuthology.orchestra.run.smithi117.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:50:30.698 INFO:teuthology.orchestra.run.smithi117.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:50:30.698 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:50:30.699 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2022-03-08T22:50:30.699 INFO:teuthology.orchestra.run.smithi117.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:50:30.699 INFO:teuthology.orchestra.run.smithi117.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:50:30.751 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-03-08T22:50:30.751 INFO:teuthology.orchestra.run.smithi117.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:50:30.752 INFO:teuthology.orchestra.run.smithi117.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:50:30.752 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:50:30.753 INFO:teuthology.orchestra.run.smithi117.stdout: thermald upower usbmuxd 2022-03-08T22:50:30.956 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:50:30.957 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:50:30.957 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:50:30.957 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:50:31.212 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:50:31.240 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:50:31.261 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:50:31.325 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:50:31.359 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:50:32.171 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:50:32.484 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:50:34.266 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:50:34.267 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:50:34.267 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:50:34.273 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:50:34.275 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:50:34.686 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 80.1 MB in 4s (22.8 MB/s) 2022-03-08T22:50:34.899 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:50:36.346 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:50:36.351 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:50:36.562 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:50:36.957 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:50:36.970 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:50:37.007 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:50:37.360 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:50:37.375 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:50:37.409 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:50:37.729 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:50:37.744 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:50:37.821 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:50:38.182 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:50:38.195 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:50:38.232 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:50:38.634 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:50:38.649 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:50:38.693 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:40.884 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:50:40.898 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:50:41.007 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:41.512 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:50:41.526 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:50:41.562 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:47.827 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:50:47.841 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:50:47.885 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:50:48.257 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:50:48.273 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:50:48.315 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:50:48.693 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package thermald. 2022-03-08T22:50:48.707 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:50:48.768 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:50:49.359 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package upower. 2022-03-08T22:50:49.374 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:50:49.427 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:50:49.893 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:50:49.906 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:50:49.951 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:50:50.275 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:50:50.397 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:50:50.557 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:50:50.683 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:50:50.834 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:50:50.960 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:50:51.102 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:50:51.566 INFO:teuthology.orchestra.run.smithi117.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:50:51.612 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:50:51.785 INFO:teuthology.orchestra.run.smithi117.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:50:51.792 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:50:51.792 INFO:teuthology.orchestra.run.smithi117.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:50:52.346 INFO:teuthology.orchestra.run.smithi117.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:50:52.566 INFO:teuthology.orchestra.run.smithi117.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:50:52.609 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:50:52.934 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:50:52.934 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:50:53.438 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:54.715 INFO:teuthology.orchestra.run.smithi117.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:50:54.715 INFO:teuthology.orchestra.run.smithi117.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:50:54.898 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:56.216 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:50:57.626 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:50:57.752 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:51:01.607 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:51:02.768 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:51:02.868 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:51:03.061 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:51:03.062 INFO:teuthology.orchestra.run.smithi117.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:51:26.610 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:51:26.611 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:51:26.621 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:51:26.639 INFO:teuthology.orchestra.run.smithi117.stdout:Generating grub configuration file ... 2022-03-08T22:51:26.961 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:51:26.983 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:51:27.397 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:51:27.409 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:51:29.993 INFO:teuthology.orchestra.run.smithi117.stdout:done 2022-03-08T22:51:30.376 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:51:30.410 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:51:30.410 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2022-03-08T22:51:30.410 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2022-03-08T22:51:30.411 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2022-03-08T22:51:30.411 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 19 2022-03-08T22:51:30.411 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:51:30.411 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2022-03-08T22:51:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:51:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:51:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:51:30.412 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:51:30.413 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2022-03-08T22:51:30.413 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2022-03-08T22:51:30.413 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:51:30.413 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2022-03-08T22:51:30.414 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:51:30.414 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:51:30.414 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:51:30.415 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:51:30.415 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:51:30.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2022-03-08T22:51:30.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2022-03-08T22:51:30.767 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:30.768 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:30.858 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:30.945 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:32.288 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:51:32.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:51:32.381 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:51:32.536 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:51:32.537 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:51:32.705 INFO:teuthology.orchestra.run.smithi117.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:51:32.706 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:51:32.707 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:51:32.728 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:51:32.728 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2022-03-08T22:51:32.728 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2022-03-08T22:51:32.729 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2022-03-08T22:51:32.729 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 19 2022-03-08T22:51:32.729 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:51:32.729 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2022-03-08T22:51:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:51:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:51:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:51:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:51:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2022-03-08T22:51:32.731 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2022-03-08T22:51:32.731 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:51:32.731 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2022-03-08T22:51:32.732 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-03-08T22:51:32.776 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.n7NLfjVWdJ 2022-03-08T22:51:32.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub/grub.cfg /tmp/tmp.n7NLfjVWdJ 2022-03-08T22:51:32.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.n7NLfjVWdJ 2022-03-08T22:51:32.878 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.n7NLfjVWdJ is 8KB 2022-03-08T22:51:32.892 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.n7NLfjVWdJ 2022-03-08T22:51:32.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:51:32.955 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:51:32.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:51:32.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:51:32.986 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:51:32.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo update-grub 2022-03-08T22:51:34.298 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:51:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:51:34.320 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2022-03-08T22:51:34.612 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:51:34.623 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:51:35.002 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:51:35.013 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:51:36.565 INFO:teuthology.orchestra.run.smithi117.stderr:done 2022-03-08T22:51:36.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2022-03-08T22:52:06.600 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:52:06.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-03-08T22:52:06.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:52:41.291 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2022-03-08T22:53:11.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-03-08T22:53:11.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:53:18.594 DEBUG:teuthology.orchestra.remote:No existing session 2022-03-08T22:53:48.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-03-08T22:53:48.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:53:48.682 DEBUG:teuthology.orchestra.run.smithi117:> true 2022-03-08T22:53:49.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-03-08T22:53:49.214 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:53:49.214 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2022-03-08T22:53:49.219 INFO:teuthology.orchestra.run.smithi117.stdout:5.13.0-35-generic 2022-03-08T22:53:49.220 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:53:49.220 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:53:49.220 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:53:50.222 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:53:50.222 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:53:50.295 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2022-03-08T22:53:50.313 DEBUG:teuthology.parallel:result is None 2022-03-08T22:53:50.314 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:53:50.322 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:53:50.323 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:53:50.331 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:53:50.368 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:53:50.402 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:53:50.402 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:53:50.422 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:53:50.434 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:53:50.435 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:53:50.458 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:53:50.466 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:53:50.467 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:53:50.476 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:53:50.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:53:50.527 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:53:50.539 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:53:50.539 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:53:50.547 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:53:50.597 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:53:50.603 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:53:50.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:53:50.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2022-03-08T22:53:50.664 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:53:50.831 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:53:50.832 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:53:50.849 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:53:50.890 DEBUG:teuthology.task.selinux:Excluding smithi117: OS 'ubuntu' does not support SELinux 2022-03-08T22:53:50.891 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:53:50.891 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:53:50.891 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:53:50.892 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:53:50.941 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:53:50.942 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:53:50.967 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:53:50.981 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T22:56:48.353 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2022-03-08T22:56:48.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-03-08T22:56:48.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:56:48.422 DEBUG:teuthology.orchestra.run.smithi117:> true 2022-03-08T22:56:48.540 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T22:56:48.551 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T22:56:48.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:56:48.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:56:48.618 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:56:48.618 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Command line: ntpd -gq 2022-03-08T22:56:48.619 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: proto: precision = 0.082 usec (-23) 2022-03-08T22:56:48.619 INFO:teuthology.orchestra.run.smithi117.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:56:48.620 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:56:48.620 INFO:teuthology.orchestra.run.smithi117.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:56:48.622 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:56:48.622 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:56:48.622 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:56:48.622 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:56:48.623 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen normally on 3 enp3s0f1 172.21.15.117:123 2022-03-08T22:56:48.623 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen normally on 4 lo [::1]:123 2022-03-08T22:56:48.623 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4463%5]:123 2022-03-08T22:56:48.623 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:48 ntpd[9039]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:56:49.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:49.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:49.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:49.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:49.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:49.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:49.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:49.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:49.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:49.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:49.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:49.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:49 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:50.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:50.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:50.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:50.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:50.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:50.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:50 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:51.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:51.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:51.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:51.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:51.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:51.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:51.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:51.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:51.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:51.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:51.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:51.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:51 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:52.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:52.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:52.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:52.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:52.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:52.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:52 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:53.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:53.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:53.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.623 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.623 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:53.623 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:53.623 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:53.624 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:53.624 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:53.624 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:53 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:54.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:54.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:54.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:54.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:54.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:54.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:54 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:55.619 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:55.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:55.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:55.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:55.620 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:55.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:55.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:55.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:55.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:55.621 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:55.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:56:55.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:56:55.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:56:55.622 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:56:55.623 INFO:teuthology.orchestra.run.smithi117.stdout: 8 Mar 22:56:55 ntpd[9039]: ntpd: time slew -0.009609 s 2022-03-08T22:56:55.623 INFO:teuthology.orchestra.run.smithi117.stdout:ntpd: time slew -0.009609s 2022-03-08T22:56:55.624 INFO:teuthology.orchestra.run.smithi117.stderr: 8 Mar 22:56:55 ntpd[9039]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:56:55.672 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:56:55.673 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2022-03-08T22:56:55.673 INFO:teuthology.orchestra.run.smithi117.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:56:55.673 INFO:teuthology.orchestra.run.smithi117.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:56:55.674 INFO:teuthology.orchestra.run.smithi117.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:56:55.674 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:56:55.675 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T22:56:55.688 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T22:56:55.689 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-08T22:56:55.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:56:55.714 INFO:teuthology.orchestra.run.smithi117.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:56:55.723 INFO:teuthology.orchestra.run.smithi117.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:56:55.730 INFO:teuthology.orchestra.run.smithi117.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:56:55.786 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:56:55.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:56:55.842 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:56:55.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:56:55.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl restart docker || true 2022-03-08T22:56:56.834 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T22:56:56.840 INFO:tasks.kubeadm:172.21.15.117 is in 172.21.0.0/20 at pos 3956 2022-03-08T22:56:56.854 INFO:tasks.kubeadm:Pod subnet: 10.251.160.0/21 2022-03-08T22:56:56.854 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T22:56:56.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:56:56.881 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:56:56.881 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:56:56.882 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:56:57.120 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:56:57.121 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:56:57.191 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:56:57.279 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:56:58.613 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:56:58.780 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:56:58.781 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:56:58.872 INFO:teuthology.orchestra.run.smithi117.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:56:58.889 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:56:58.889 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:56:58.890 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:56:58.943 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:56:59.115 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:56:59.116 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:56:59.287 INFO:teuthology.orchestra.run.smithi117.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:56:59.287 INFO:teuthology.orchestra.run.smithi117.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:56:59.288 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-03-08T22:56:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: apt-transport-https 2022-03-08T22:56:59.476 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:56:59.476 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 4,680 B of archives. 2022-03-08T22:56:59.476 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:56:59.477 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:56:59.698 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:56:59.698 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:56:59.698 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:56:59.716 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:56:59.716 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:56:59.716 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:56:59.722 INFO:teuthology.orchestra.run.smithi117.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:56:59.772 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 4,680 B in 0s (25.9 kB/s) 2022-03-08T22:56:59.840 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:56:59.883 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:56:59.887 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:56:59.922 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:57:00.201 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:57:01.543 INFO:teuthology.orchestra.run.smithi117.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:57:01.558 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:57:01.558 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:57:01.558 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:57:01.689 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:57:01.802 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:57:01.884 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:57:01.972 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:57:02.028 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:57:02.383 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:57:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 63.3 kB in 1s (78.7 kB/s) 2022-03-08T22:57:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:57:03.527 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:57:03.527 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:57:03.620 INFO:teuthology.orchestra.run.smithi117.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:57:03.636 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:57:03.636 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:57:03.636 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:57:03.693 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T22:57:03.868 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T22:57:03.869 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T22:57:04.018 INFO:teuthology.orchestra.run.smithi117.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:57:04.018 INFO:teuthology.orchestra.run.smithi117.stdout:bridge-utils set to manually installed. 2022-03-08T22:57:04.019 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2022-03-08T22:57:04.019 INFO:teuthology.orchestra.run.smithi117.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:57:04.020 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2022-03-08T22:57:04.020 INFO:teuthology.orchestra.run.smithi117.stdout: nftables 2022-03-08T22:57:04.070 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-03-08T22:57:04.071 INFO:teuthology.orchestra.run.smithi117.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:57:04.266 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:57:04.266 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:57:04.266 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:57:04.266 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:57:04.316 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:57:04.439 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:57:04.546 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:57:04.591 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:57:05.141 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:57:05.729 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:57:05.958 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:57:06.345 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:57:06.345 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:57:06.345 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:57:06.353 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:57:06.353 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:57:06.354 INFO:teuthology.orchestra.run.smithi117.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:57:06.359 INFO:teuthology.orchestra.run.smithi117.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:57:06.411 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 73.6 MB in 2s (36.1 MB/s) 2022-03-08T22:57:06.496 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:57:06.538 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:57:06.541 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:57:06.586 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:57:06.880 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:57:06.888 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:57:06.930 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:57:07.983 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:57:07.996 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:57:08.057 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:57:08.319 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:57:08.334 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:57:08.377 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:57:09.854 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package socat. 2022-03-08T22:57:09.869 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:57:09.912 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:57:10.257 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:57:10.270 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:57:10.310 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:57:12.688 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:57:12.701 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:57:12.729 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:57:13.898 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:57:13.913 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:57:13.956 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:57:15.104 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:57:15.229 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:57:15.355 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:57:16.128 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:57:16.254 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:57:16.380 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:57:16.506 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:57:16.721 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:57:17.224 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:57:17.617 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:57:18.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:57:32.039 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:57:34.941 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:57:37.121 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:57:40.896 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:57:42.345 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:57:47.642 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:57:49.702 INFO:teuthology.orchestra.run.smithi117.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:57:49.706 INFO:tasks.kubeadm:Token: abcdef.xnd9y02x0anm9kun 2022-03-08T22:57:49.707 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): '172.21.15.117'} 2022-03-08T22:57:49.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo kubeadm init --node-name smithi117 --token abcdef.xnd9y02x0anm9kun --pod-network-cidr 10.251.160.0/21 2022-03-08T22:57:49.908 INFO:teuthology.orchestra.run.smithi117.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T22:57:49.909 INFO:teuthology.orchestra.run.smithi117.stdout:[preflight] Running pre-flight checks 2022-03-08T22:57:50.291 INFO:teuthology.orchestra.run.smithi117.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T22:57:50.292 INFO:teuthology.orchestra.run.smithi117.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T22:57:50.292 INFO:teuthology.orchestra.run.smithi117.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T22:57:50.594 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T22:57:50.667 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "ca" certificate and key 2022-03-08T22:57:50.795 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T22:57:50.795 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi117] and IPs [10.96.0.1 172.21.15.117] 2022-03-08T22:57:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T22:57:51.270 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T22:57:51.479 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T22:57:51.740 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T22:57:51.861 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T22:57:51.862 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi117] and IPs [172.21.15.117 127.0.0.1 ::1] 2022-03-08T22:57:52.245 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T22:57:52.245 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi117] and IPs [172.21.15.117 127.0.0.1 ::1] 2022-03-08T22:57:52.431 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T22:57:52.685 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T22:57:52.815 INFO:teuthology.orchestra.run.smithi117.stdout:[certs] Generating "sa" key and public key 2022-03-08T22:57:52.815 INFO:teuthology.orchestra.run.smithi117.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T22:57:53.044 INFO:teuthology.orchestra.run.smithi117.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T22:57:53.167 INFO:teuthology.orchestra.run.smithi117.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T22:57:53.271 INFO:teuthology.orchestra.run.smithi117.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T22:57:53.437 INFO:teuthology.orchestra.run.smithi117.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T22:57:53.452 INFO:teuthology.orchestra.run.smithi117.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:57:53.454 INFO:teuthology.orchestra.run.smithi117.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:57:53.454 INFO:teuthology.orchestra.run.smithi117.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:57:53.682 INFO:teuthology.orchestra.run.smithi117.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T22:57:53.682 INFO:teuthology.orchestra.run.smithi117.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T22:57:53.685 INFO:teuthology.orchestra.run.smithi117.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T22:57:53.686 INFO:teuthology.orchestra.run.smithi117.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T22:57:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T22:57:53.689 INFO:teuthology.orchestra.run.smithi117.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T22:58:11.222 INFO:teuthology.orchestra.run.smithi117.stdout:[apiclient] All control plane components are healthy after 17.533103 seconds 2022-03-08T22:58:11.223 INFO:teuthology.orchestra.run.smithi117.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T22:58:11.258 INFO:teuthology.orchestra.run.smithi117.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T22:58:11.259 INFO:teuthology.orchestra.run.smithi117.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T22:58:11.811 INFO:teuthology.orchestra.run.smithi117.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T22:58:11.811 INFO:teuthology.orchestra.run.smithi117.stdout:[mark-control-plane] Marking the node smithi117 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T22:58:11.812 INFO:teuthology.orchestra.run.smithi117.stdout:[mark-control-plane] Marking the node smithi117 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T22:58:12.325 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] Using token: abcdef.xnd9y02x0anm9kun 2022-03-08T22:58:12.325 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T22:58:12.347 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T22:58:12.366 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T22:58:12.375 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T22:58:12.383 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T22:58:12.455 INFO:teuthology.orchestra.run.smithi117.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T22:58:12.492 INFO:teuthology.orchestra.run.smithi117.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T22:58:12.964 INFO:teuthology.orchestra.run.smithi117.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T22:58:13.113 INFO:teuthology.orchestra.run.smithi117.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T22:58:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.114 INFO:teuthology.orchestra.run.smithi117.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T22:58:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.115 INFO:teuthology.orchestra.run.smithi117.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T22:58:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: mkdir -p $HOME/.kube 2022-03-08T22:58:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T22:58:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T22:58:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.116 INFO:teuthology.orchestra.run.smithi117.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T22:58:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T22:58:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.118 INFO:teuthology.orchestra.run.smithi117.stdout:You should now deploy a pod network to the cluster. 2022-03-08T22:58:13.118 INFO:teuthology.orchestra.run.smithi117.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T22:58:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T22:58:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.119 INFO:teuthology.orchestra.run.smithi117.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T22:58:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T22:58:13.119 INFO:teuthology.orchestra.run.smithi117.stdout:kubeadm join 172.21.15.117:6443 --token abcdef.xnd9y02x0anm9kun \ 2022-03-08T22:58:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: --discovery-token-ca-cert-hash sha256:e997d48c7f7a2cc433d3a2f24edf83ef33e0e9418aeaeb50e72762c60746ffe7 2022-03-08T22:58:13.122 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:13.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T22:58:13.132 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T22:58:13.133 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:58:13.189 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:13.189 DEBUG:teuthology.orchestra.run.smithi117:> dd of=.kube/config 2022-03-08T22:58:13.195 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:13.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/root/.kube/config 2022-03-08T22:58:13.206 DEBUG:teuthology.orchestra.run.smithi117:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T22:58:15.833 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T22:58:15.884 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T22:58:15.954 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T22:58:16.036 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T22:58:16.130 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T22:58:16.179 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T22:58:16.297 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T22:58:16.304 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T22:58:16.355 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T22:58:16.444 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T22:58:16.476 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T22:58:16.527 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T22:58:16.635 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T22:58:16.686 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T22:58:16.786 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T22:58:16.931 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T22:58:16.937 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T22:58:16.986 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T22:58:17.094 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T22:58:17.158 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T22:58:17.270 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T22:58:17.303 INFO:teuthology.orchestra.run.smithi117.stdout:namespace/tigera-operator created 2022-03-08T22:58:17.314 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:58:17.319 INFO:teuthology.orchestra.run.smithi117.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T22:58:17.339 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/tigera-operator created 2022-03-08T22:58:17.351 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:58:17.444 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:58:17.463 INFO:teuthology.orchestra.run.smithi117.stdout:deployment.apps/tigera-operator created 2022-03-08T22:58:17.469 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f - 2022-03-08T22:58:20.241 INFO:teuthology.orchestra.run.smithi117.stdout:installation.operator.tigera.io/default created 2022-03-08T22:58:20.247 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T22:58:20.248 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:20.248 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:58:20.254 INFO:tasks.kubeadm:Scratch devices on smithi117: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:58:20.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:58:20.326 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records in 2022-03-08T22:58:20.326 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2022-03-08T22:58:20.327 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0187039 s, 561 MB/s 2022-03-08T22:58:20.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:58:20.354 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records in 2022-03-08T22:58:20.354 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2022-03-08T22:58:20.355 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137803 s, 761 MB/s 2022-03-08T22:58:20.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:58:20.427 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records in 2022-03-08T22:58:20.428 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2022-03-08T22:58:20.428 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156623 s, 669 MB/s 2022-03-08T22:58:20.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:58:20.456 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records in 2022-03-08T22:58:20.456 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2022-03-08T22:58:20.457 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154464 s, 679 MB/s 2022-03-08T22:58:20.464 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T22:58:20.465 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi117-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi117 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi117-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi117 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi117-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi117 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi117-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi117 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T22:58:20.465 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f - 2022-03-08T22:58:21.310 INFO:teuthology.orchestra.run.smithi117.stdout:persistentvolume/smithi117-lv-1 created 2022-03-08T22:58:21.313 INFO:teuthology.orchestra.run.smithi117.stdout:persistentvolume/smithi117-lv-2 created 2022-03-08T22:58:21.322 INFO:teuthology.orchestra.run.smithi117.stdout:persistentvolume/smithi117-lv-3 created 2022-03-08T22:58:21.330 INFO:teuthology.orchestra.run.smithi117.stdout:persistentvolume/smithi117-lv-4 created 2022-03-08T22:58:21.358 INFO:teuthology.orchestra.run.smithi117.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T22:58:21.363 DEBUG:teuthology.orchestra.run.smithi117:> kubectl taint node smithi117 node-role.kubernetes.io/master- || true 2022-03-08T22:58:21.432 INFO:teuthology.orchestra.run.smithi117.stdout:node/smithi117 untainted 2022-03-08T22:58:21.435 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T22:58:21.435 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T22:58:21.447 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T22:58:21.448 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:21.449 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:58:21.455 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:58:21.456 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:58:21.505 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:58:21.505 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:58:21.505 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T22:58:21.505 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:58:21.506 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-03-08 22:58:20.334470851 +0000 2022-03-08T22:58:21.506 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-03-08 22:58:20.334470851 +0000 2022-03-08T22:58:21.506 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-03-08 22:58:20.334470851 +0000 2022-03-08T22:58:21.506 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-03-08T22:58:21.512 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-03-08T22:58:21.513 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-03-08T22:58:21.513 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000544176 s, 941 kB/s 2022-03-08T22:58:21.516 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:58:21.520 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:58:21.521 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:58:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-08T22:58:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:58:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-03-08 22:58:20.354470851 +0000 2022-03-08T22:58:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-03-08 22:58:20.354470851 +0000 2022-03-08T22:58:21.522 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-03-08 22:58:20.354470851 +0000 2022-03-08T22:58:21.522 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-03-08T22:58:21.528 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-03-08T22:58:21.528 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-03-08T22:58:21.528 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000469035 s, 1.1 MB/s 2022-03-08T22:58:21.530 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:58:21.535 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:58:21.535 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:58:21.536 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-08T22:58:21.536 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:58:21.536 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-03-08 22:58:20.430470849 +0000 2022-03-08T22:58:21.536 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-03-08 22:58:20.430470849 +0000 2022-03-08T22:58:21.537 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-03-08 22:58:20.430470849 +0000 2022-03-08T22:58:21.537 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-03-08T22:58:21.540 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-03-08T22:58:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-03-08T22:58:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000473251 s, 1.1 MB/s 2022-03-08T22:58:21.544 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:58:21.549 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:58:21.550 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:58:21.550 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-08T22:58:21.550 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:58:21.551 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-03-08 22:58:20.458470849 +0000 2022-03-08T22:58:21.551 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-03-08 22:58:20.458470849 +0000 2022-03-08T22:58:21.551 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-03-08 22:58:20.458470849 +0000 2022-03-08T22:58:21.551 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-03-08T22:58:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-03-08T22:58:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-03-08T22:58:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000487467 s, 1.1 MB/s 2022-03-08T22:58:21.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:58:21.722 INFO:teuthology.orchestra.run.smithi117.stdout:loop 2022-03-08T22:58:21.723 INFO:tasks.nvme_loop:Connecting nvme_loop smithi117:/dev/vg_nvme/lv_1... 2022-03-08T22:58:21.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:58:21.739 INFO:teuthology.orchestra.run.smithi117.stdout:1 2022-03-08T22:58:21.759 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:58:21.776 INFO:tasks.nvme_loop:Connecting nvme_loop smithi117:/dev/vg_nvme/lv_2... 2022-03-08T22:58:21.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:58:21.836 INFO:teuthology.orchestra.run.smithi117.stdout:1 2022-03-08T22:58:21.855 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:58:21.871 INFO:tasks.nvme_loop:Connecting nvme_loop smithi117:/dev/vg_nvme/lv_3... 2022-03-08T22:58:21.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:58:21.889 INFO:teuthology.orchestra.run.smithi117.stdout:1 2022-03-08T22:58:21.914 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:58:21.929 INFO:tasks.nvme_loop:Connecting nvme_loop smithi117:/dev/vg_nvme/lv_4... 2022-03-08T22:58:21.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:58:21.989 INFO:teuthology.orchestra.run.smithi117.stdout:1 2022-03-08T22:58:22.011 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:58:22.033 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:22.034 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:58:22.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo nvme list 2022-03-08T22:58:22.051 INFO:teuthology.orchestra.run.smithi117.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:58:22.051 INFO:teuthology.orchestra.run.smithi117.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:58:22.051 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme0n1 PHFT620400V6400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T22:58:22.052 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme1n1 4e071d38231f8415 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:22.052 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme2n1 fe1d42baf42464e5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:22.052 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme3n1 fd52825210d3e580 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:22.053 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-08T22:58:23.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo nvme list 2022-03-08T22:58:23.070 INFO:teuthology.orchestra.run.smithi117.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:58:23.070 INFO:teuthology.orchestra.run.smithi117.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:58:23.070 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme0n1 PHFT620400V6400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T22:58:23.071 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme1n1 4e071d38231f8415 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:23.071 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme2n1 fe1d42baf42464e5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:23.071 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme3n1 fd52825210d3e580 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:23.071 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/nvme4n1 f6b6ee3012b9f3cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:58:23.073 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:58:23.073 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:23.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/scratch_devs 2022-03-08T22:58:23.084 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T22:58:23.155 INFO:tasks.rook:Rook start 2022-03-08T22:58:23.155 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-08T22:58:23.156 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T22:58:23.156 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-08T22:58:23.156 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-08T22:58:23.163 INFO:teuthology.orchestra.run.smithi117.stderr:Cloning into 'rook'... 2022-03-08T22:58:26.303 INFO:teuthology.orchestra.run.smithi117.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-08T22:58:26.303 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.303 INFO:teuthology.orchestra.run.smithi117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-08T22:58:26.304 INFO:teuthology.orchestra.run.smithi117.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-08T22:58:26.304 INFO:teuthology.orchestra.run.smithi117.stderr:state without impacting any branches by switching back to a branch. 2022-03-08T22:58:26.304 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.304 INFO:teuthology.orchestra.run.smithi117.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-08T22:58:26.305 INFO:teuthology.orchestra.run.smithi117.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-08T22:58:26.305 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.305 INFO:teuthology.orchestra.run.smithi117.stderr: git switch -c 2022-03-08T22:58:26.305 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.306 INFO:teuthology.orchestra.run.smithi117.stderr:Or undo this operation with: 2022-03-08T22:58:26.306 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.306 INFO:teuthology.orchestra.run.smithi117.stderr: git switch - 2022-03-08T22:58:26.307 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.307 INFO:teuthology.orchestra.run.smithi117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-08T22:58:26.307 INFO:teuthology.orchestra.run.smithi117.stderr: 2022-03-08T22:58:26.407 INFO:tasks.rook:/home/teuthworker 2022-03-08T22:58:26.407 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T22:58:26.416 INFO:teuthology.orchestra.run.smithi117.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T22:58:26.416 INFO:teuthology.orchestra.run.smithi117.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T22:58:26.416 INFO:teuthology.orchestra.run.smithi117.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T22:58:26.417 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:58:26.417 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 ::1/128 scope host 2022-03-08T22:58:26.417 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:58:26.417 INFO:teuthology.orchestra.run.smithi117.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:58:26.417 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8a brd ff:ff:ff:ff:ff:ff 2022-03-08T22:58:26.418 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f0 2022-03-08T22:58:26.418 INFO:teuthology.orchestra.run.smithi117.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:58:26.418 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8b brd ff:ff:ff:ff:ff:ff 2022-03-08T22:58:26.418 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f1 2022-03-08T22:58:26.418 INFO:teuthology.orchestra.run.smithi117.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:58:26.419 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:62 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:58:26.419 INFO:teuthology.orchestra.run.smithi117.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T22:58:26.419 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:63 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:58:26.419 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.21.15.117/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T22:58:26.420 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft 42893sec preferred_lft 42893sec 2022-03-08T22:58:26.420 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 fe80::ec4:7aff:fe8f:4463/64 scope link 2022-03-08T22:58:26.420 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:58:26.420 INFO:teuthology.orchestra.run.smithi117.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T22:58:26.421 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 02:42:49:2a:12:a4 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:58:26.421 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T22:58:26.421 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:58:26.422 DEBUG:teuthology.orchestra.run.smithi117:> which systemctl 2022-03-08T22:58:26.427 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/systemctl 2022-03-08T22:58:26.427 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T22:58:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T22:58:26.475 DEBUG:teuthology.orchestra.run.smithi117:> true 2022-03-08T22:58:26.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-03-08T22:58:26.547 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-03-08T22:58:26.552 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:58:26.553 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T22:58:26.610 DEBUG:teuthology.orchestra.remote:smithi117:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:58:26.661 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:26.662 DEBUG:teuthology.orchestra.run.smithi117:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-08T22:58:26.671 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-08T22:58:26.711 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T22:58:26.711 DEBUG:teuthology.orchestra.run.smithi117:> dd of=operator.yaml 2022-03-08T22:58:26.719 INFO:tasks.rook:Deploying operator 2022-03-08T22:58:26.783 DEBUG:teuthology.orchestra.remote:smithi117:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:58:26.815 DEBUG:teuthology.orchestra.remote:smithi117:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:58:26.829 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-08T22:58:27.416 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T22:58:27.606 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T22:58:27.924 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T22:58:27.973 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T22:58:28.098 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T22:58:28.120 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T22:58:28.152 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T22:58:28.289 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T22:58:28.292 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T22:58:28.341 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T22:58:28.493 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T22:58:28.526 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T22:58:28.565 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T22:58:28.685 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T22:58:28.692 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-08T22:58:28.716 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-08T22:58:28.759 INFO:teuthology.orchestra.run.smithi117.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-08T22:58:28.884 INFO:teuthology.orchestra.run.smithi117.stdout:namespace/rook-ceph created 2022-03-08T22:58:28.918 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:58:28.948 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-08T22:58:29.034 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-08T22:58:29.060 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-08T22:58:29.085 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:58:29.119 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:58:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:58:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:58:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:58:29.585 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:58:29.587 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-system created 2022-03-08T22:58:29.621 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:58:29.747 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:58:29.822 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:58:29.872 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T22:58:29.874 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T22:58:29.954 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T22:58:30.064 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:58:30.124 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:58:30.241 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:58:30.244 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:58:30.272 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:58:30.288 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:58:30.305 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:58:30.330 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:58:30.401 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:58:30.404 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:58:30.420 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:58:30.438 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:58:30.462 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:58:30.463 INFO:teuthology.orchestra.run.smithi117.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T22:58:30.534 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T22:58:30.545 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T22:58:30.553 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T22:58:30.562 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T22:58:30.570 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T22:58:30.593 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T22:58:30.604 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T22:58:30.612 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T22:58:30.620 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T22:58:30.727 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T22:58:30.778 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:58:30.787 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T22:58:30.810 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T22:58:30.820 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T22:58:30.876 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:58:30.880 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T22:58:30.887 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T22:58:30.896 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T22:58:30.912 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T22:58:30.941 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T22:58:31.063 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:58:31.070 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T22:58:31.079 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T22:58:31.087 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T22:58:31.113 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:58:31.120 INFO:teuthology.orchestra.run.smithi117.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T22:58:31.129 INFO:teuthology.orchestra.run.smithi117.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:58:31.137 INFO:teuthology.orchestra.run.smithi117.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:58:31.145 INFO:teuthology.orchestra.run.smithi117.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T22:58:31.154 INFO:teuthology.orchestra.run.smithi117.stdout:configmap/rook-ceph-operator-config created 2022-03-08T22:58:31.230 INFO:teuthology.orchestra.run.smithi117.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T22:58:31.234 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:58:31.399 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:58:31.399 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 Pending 0 0s 2022-03-08T22:58:41.404 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:58:41.470 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:58:41.471 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 Pending 0 10s 2022-03-08T22:58:51.474 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:58:51.556 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:58:51.557 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 Pending 0 20s 2022-03-08T22:59:01.559 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:01.788 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:01.788 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 Pending 0 30s 2022-03-08T22:59:11.791 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:11.847 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:11.847 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 Pending 0 40s 2022-03-08T22:59:21.850 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:21.918 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:21.919 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 ContainerCreating 0 50s 2022-03-08T22:59:31.922 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:31.990 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:31.991 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 ContainerCreating 0 60s 2022-03-08T22:59:41.994 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:42.253 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:42.254 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 ContainerCreating 0 71s 2022-03-08T22:59:52.257 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:59:52.320 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:59:52.320 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 0/1 ContainerCreating 0 81s 2022-03-08T23:00:02.323 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:00:02.387 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:02.387 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-operator-6b496c8959-grrt9 1/1 Running 0 91s 2022-03-08T23:00:02.389 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-grrt9 2022-03-08T23:00:02.396 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:00:02.397 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:02.450 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-08T23:00:02.451 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T23:00:02.451 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T23:00:02.452 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T23:00:02.452 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T23:00:02.452 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T23:00:02.453 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T23:00:02.453 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T23:00:02.454 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T23:00:02.456 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f - 2022-03-08T23:00:02.487 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.234568 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T23:00:02.488 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.234654 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-08T23:00:02.488 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.234657 I | cephcmd: starting Rook-Ceph operator 2022-03-08T23:00:02.488 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.370620 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-08T23:00:02.489 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.397186 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:00:02.489 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.399184 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:00:02.489 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.401040 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-08T23:00:02.489 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.402396 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:00:02.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.404444 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T23:00:02.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.404461 I | operator: watching all namespaces for ceph cluster CRs 2022-03-08T23:00:02.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.404578 I | operator: setting up the controller-runtime manager 2022-03-08T23:00:02.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.406017 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-08T23:00:02.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.409295 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:00:02.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.811532 I | ceph-cluster-controller: successfully started 2022-03-08T23:00:02.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.811636 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:00:02.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.811705 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:00:02.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.811815 I | ceph-block-pool-controller: successfully started 2022-03-08T23:00:02.492 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.811914 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:00:02.492 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.812002 I | ceph-object-realm-controller: successfully started 2022-03-08T23:00:02.492 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.812081 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:00:02.492 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.812891 I | ceph-object-zone-controller: successfully started 2022-03-08T23:00:02.492 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.813230 I | ceph-object-controller: successfully started 2022-03-08T23:00:02.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.813431 I | ceph-file-controller: successfully started 2022-03-08T23:00:02.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.813582 I | ceph-nfs-controller: successfully started 2022-03-08T23:00:02.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.813725 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:00:02.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.813886 I | ceph-client-controller: successfully started 2022-03-08T23:00:02.494 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.814001 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:00:02.494 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 22:59:54.815299 I | operator: starting the controller-runtime manager 2022-03-08T23:00:03.421 INFO:teuthology.orchestra.run.smithi117.stdout:configmap/rook-config-override created 2022-03-08T23:00:03.425 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-03-08T23:00:03.426 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:00:03.426 DEBUG:teuthology.orchestra.run.smithi117:> dd of=cluster.yaml 2022-03-08T23:00:03.431 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f cluster.yaml 2022-03-08T23:00:05.875 INFO:teuthology.orchestra.run.smithi117.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T23:00:05.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.884474 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-08T23:00:05.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.884594 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T23:00:05.933 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.936128 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:00:05.933 INFO:tasks.rook.operator.smithi117.stdout:   ... // 3 identical fields 2022-03-08T23:00:05.933 INFO:tasks.rook.operator.smithi117.stdout:   Labels: nil, 2022-03-08T23:00:05.939 INFO:tasks.rook.operator.smithi117.stdout:   Placement: nil, 2022-03-08T23:00:05.939 INFO:tasks.rook.operator.smithi117.stdout:   Network: v1.NetworkSpec{ 2022-03-08T23:00:05.940 INFO:tasks.rook.operator.smithi117.stdout:   Provider: "", 2022-03-08T23:00:05.940 INFO:tasks.rook.operator.smithi117.stdout:   Selectors: nil, 2022-03-08T23:00:05.940 INFO:tasks.rook.operator.smithi117.stdout:   HostNetwork: false, 2022-03-08T23:00:05.940 INFO:tasks.rook.operator.smithi117.stdout:-  IPFamily: "", 2022-03-08T23:00:05.941 INFO:tasks.rook.operator.smithi117.stdout:+  IPFamily: "IPv4", 2022-03-08T23:00:05.941 INFO:tasks.rook.operator.smithi117.stdout:   DualStack: false, 2022-03-08T23:00:05.941 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:00:05.941 INFO:tasks.rook.operator.smithi117.stdout:   Resources: nil, 2022-03-08T23:00:05.942 INFO:tasks.rook.operator.smithi117.stdout:   PriorityClassNames: nil, 2022-03-08T23:00:05.942 INFO:tasks.rook.operator.smithi117.stdout:   ... // 16 identical fields 2022-03-08T23:00:05.942 INFO:tasks.rook.operator.smithi117.stdout:  } 2022-03-08T23:00:05.942 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.936629 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:05.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.941550 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T23:00:05.944 DEBUG:teuthology.orchestra.remote:smithi117:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T23:00:05.950 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.954007 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:00:05.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.957771 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:00:05.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.957839 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-08T23:00:05.956 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.959811 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:00:05.958 DEBUG:teuthology.orchestra.run.smithi117:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T23:00:05.961 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.961789 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:00:05.961 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.963397 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:00:05.962 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.965412 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:00:05.965 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.967342 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-08T23:00:05.969 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:05.972804 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:00:06.136 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:06.140110 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-08T23:00:06.217 INFO:teuthology.orchestra.run.smithi117.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T23:00:06.222 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T23:00:06.223 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:06.367 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:06.368 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 Pending 0 0s 2022-03-08T23:00:06.605 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:06.608632 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-08T23:00:07.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.398947 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-08T23:00:07.421 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.424840 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-08T23:00:07.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.427139 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:00:07.546 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.549629 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-08T23:00:07.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.749733 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:00:07.937 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.940282 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:00:07.937 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:07.940318 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-08T23:00:08.136 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:08.139241 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:00:11.370 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:11.428 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:11.428 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 5s 2022-03-08T23:00:16.431 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:16.501 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:16.502 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 10s 2022-03-08T23:00:21.505 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:21.573 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:21.574 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 15s 2022-03-08T23:00:26.576 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:26.647 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:26.647 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 20s 2022-03-08T23:00:31.652 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:31.753 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:31.753 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 25s 2022-03-08T23:00:36.756 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:37.935 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:37.935 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 31s 2022-03-08T23:00:42.938 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:42.989 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:42.990 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 36s 2022-03-08T23:00:45.116 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.124312 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-08T23:00:45.168 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.177913 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:00:45.566 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.576071 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:00:45.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.579646 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:00:45.589 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.599355 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:00:45.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.623527 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:00:45.971 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:45.981182 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:00:46.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.181135 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:00:46.179 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.187159 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:00:46.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.187189 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:00:46.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.187201 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:00:46.200 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.209368 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:00:46.238 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.247306 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:00:46.239 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.247822 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:00:46.244 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.251041 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:00:46.346 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.356322 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:00:46.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.507479 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:00:46.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.958372 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:00:46.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:46.958404 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:00:47.140 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:47.150674 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-08T23:00:47.143 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:47.151071 I | op-mon: start running mons 2022-03-08T23:00:47.297 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:47.307394 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:00:47.571 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:47.580870 I | op-mon: creating mon secrets for a new cluster 2022-03-08T23:00:47.992 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:48.086 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:48.086 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 42s 2022-03-08T23:00:48.113 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:48.124309 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:00:48.499 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:48.509541 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:00:48.699 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:48.709067 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T23:00:48.899 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:48.909519 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:00:49.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:49.112150 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T23:00:49.298 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:49.308992 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:00:49.698 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:49.708331 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:00:50.099 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:50.109575 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:00:50.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:50.508252 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:00:50.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:50.763606 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:00:50.754 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:50.763762 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:00:50.898 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:50.908298 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:00:51.466 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:51.476852 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:00:51.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:51.802915 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:00:51.808 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:51.818170 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:00:52.682 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:52.692868 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:00:52.747 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:52.757776 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:00:53.090 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:53.212 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.221808 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:00:53.214 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:53.215 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 47s 2022-03-08T23:00:53.353 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.363699 I | op-mon: targeting the mon count 1 2022-03-08T23:00:53.440 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.450414 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T23:00:53.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.450531 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:53.526 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.535946 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:00:53.897 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:53.907929 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:00:54.096 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:54.106748 I | op-mon: parsing mon endpoints: 2022-03-08T23:00:54.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:54.106768 W | op-mon: ignoring invalid monitor 2022-03-08T23:00:54.105 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:54.115650 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:00:54.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:54.451496 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:00:54.449 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:54.459989 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:00:55.198 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:55.209025 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:00:55.271 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:55.281901 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:00:55.338 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:55.346524 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-08T23:00:55.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:00:55.355203 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:00:58.220 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:00:58.277 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:58.278 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 52s 2022-03-08T23:01:03.281 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:03.354 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:03.355 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 57s 2022-03-08T23:01:07.306 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.315932 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:01:07.306 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.315961 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:07.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.315972 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:01:07.312 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.320769 I | op-mon: parsing mon endpoints: 2022-03-08T23:01:07.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.320797 W | op-mon: ignoring invalid monitor 2022-03-08T23:01:07.381 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.390861 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:07.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.392535 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:07.501 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.511754 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:01:07.502 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:01:07.504+0000 7f1d12b48700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:01:07.502 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:01:07.504+0000 7f1d12b48700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:01:07.502 INFO:tasks.rook.operator.smithi117.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-08T23:01:07.502 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.511777 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:01:07.842 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.852908 I | op-mon: start running mons 2022-03-08T23:01:07.846 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.856878 I | op-mon: parsing mon endpoints: 2022-03-08T23:01:07.846 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.856895 W | op-mon: ignoring invalid monitor 2022-03-08T23:01:07.880 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.890903 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T23:01:07.922 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.932188 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:07.922 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:07.932429 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:08.357 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:08.629 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:08.630 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 62s 2022-03-08T23:01:09.005 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:09.014474 I | op-mon: targeting the mon count 1 2022-03-08T23:01:09.011 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:09.022776 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T23:01:09.255 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:09.266183 I | op-mon: sched-mon: waiting for canary pod creation rook-ceph-mon-a-canary 2022-03-08T23:01:13.632 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:13.693 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:13.693 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 0/1 ContainerCreating 0 67s 2022-03-08T23:01:14.287 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:14.297448 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi117 2022-03-08T23:01:14.287 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:14.297474 I | op-mon: assignmon: mon a assigned to node smithi117 2022-03-08T23:01:14.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:14.318240 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T23:01:14.542 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:14.529885 I | op-mon: creating mon a 2022-03-08T23:01:15.555 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:15.566027 I | op-mon: mon "a" endpoint is [v2:10.106.34.234:3300,v1:10.106.34.234:6789] 2022-03-08T23:01:15.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:15.864629 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:01:15.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:15.864720 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:-1] 2022-03-08T23:01:15.856 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:15.865105 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:01:16.474 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:16.485144 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:16.477 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:16.485540 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:16.501 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:16.511945 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T23:01:16.691 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:16.702108 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T23:01:17.673 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:17.683848 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:01:17.673 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:17.683882 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:01:17.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:17.690195 I | op-mon: mon a is not yet running 2022-03-08T23:01:17.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:17.690222 I | op-mon: mons running: [] 2022-03-08T23:01:18.696 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:19.009 INFO:teuthology.orchestra.run.smithi117.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:19.010 INFO:teuthology.orchestra.run.smithi117.stdout:rook-ceph-tools-7677f7f4b9-5xdk6 1/1 Running 0 73s 2022-03-08T23:01:19.011 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T23:01:19.012 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:01:19.280 INFO:teuthology.orchestra.run.smithi117.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:01:19.280 INFO:teuthology.orchestra.run.smithi117.stderr:2022-03-08T23:01:19.288+0000 7f653d405700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:01:19.280 INFO:teuthology.orchestra.run.smithi117.stderr:2022-03-08T23:01:19.288+0000 7f653d405700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:01:19.281 INFO:teuthology.orchestra.run.smithi117.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:01:19.282 INFO:teuthology.orchestra.run.smithi117.stderr:command terminated with exit code 1 2022-03-08T23:01:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T23:01:29.286 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:01:34.542 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:01:34.542 INFO:teuthology.orchestra.run.smithi117.stderr:no valid command found; 10 closest matches: 2022-03-08T23:01:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:pg stat 2022-03-08T23:01:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:pg getmap 2022-03-08T23:01:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:01:34.543 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:01:34.544 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_pools_json 2022-03-08T23:01:34.544 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-pool [...] 2022-03-08T23:01:34.544 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-primary [] [...] 2022-03-08T23:01:34.544 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-osd [] [...] 2022-03-08T23:01:34.545 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls [] [...] 2022-03-08T23:01:34.545 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:01:34.545 INFO:teuthology.orchestra.run.smithi117.stderr:Error EINVAL: invalid command 2022-03-08T23:01:34.545 INFO:teuthology.orchestra.run.smithi117.stderr:command terminated with exit code 22 2022-03-08T23:01:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:01:37.794 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:37.805652 I | op-mon: mons running: [a] 2022-03-08T23:01:38.020 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.031638 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:01:38.021 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.031673 I | op-mon: mons created: 1 2022-03-08T23:01:38.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.258991 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:01:38.253 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.264801 I | op-mon: mons running: [a] 2022-03-08T23:01:38.471 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.482264 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:01:38.471 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.482292 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:38.767 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.778273 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:38.768 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:38.778306 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:39.050 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.061538 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:39.051 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.061565 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:39.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.378876 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:39.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.378911 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:39.643 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.653877 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:39.643 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.653915 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:39.924 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.935425 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:39.924 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:39.935457 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:01:40.148 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:40.159676 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:01:40.149 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:40.159709 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:01:40.536 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:40.547287 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:01:40.827 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:40.838258 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:01:41.118 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:41.129081 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:01:41.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:41.575083 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:01:41.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:41.575103 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:01:41.867 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:41.877795 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:01:42.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.097175 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:01:42.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.097221 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:01:42.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.097229 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:01:42.362 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.373458 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:01:42.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.407443 I | op-mgr: start running mgr 2022-03-08T23:01:42.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:42.407495 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:01:44.550 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:01:45.026 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:01:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:no valid command found; 10 closest matches: 2022-03-08T23:01:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:pg stat 2022-03-08T23:01:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:pg getmap 2022-03-08T23:01:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:01:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:01:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_pools_json 2022-03-08T23:01:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-pool [...] 2022-03-08T23:01:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-primary [] [...] 2022-03-08T23:01:45.029 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-osd [] [...] 2022-03-08T23:01:45.029 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls [] [...] 2022-03-08T23:01:45.029 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:01:45.029 INFO:teuthology.orchestra.run.smithi117.stderr:Error EINVAL: invalid command 2022-03-08T23:01:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:command terminated with exit code 22 2022-03-08T23:01:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:01:52.013 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.024603 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T23:01:52.017 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.027518 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:01:52.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.097330 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:01:52.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.097352 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T23:01:52.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.097361 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:01:52.112 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.123686 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:01:52.117 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.128858 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T23:01:52.117 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.128883 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:01:52.117 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.128889 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:01:52.417 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.429026 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:01:52.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.429043 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:52.818 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.437038 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:01:52.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.437076 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:01:52.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.437105 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:01:52.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.442943 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-08T23:01:52.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.457504 I | op-mgr: successful modules: dashboard 2022-03-08T23:01:52.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.475851 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:01:52.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.475868 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:52.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.476842 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-08T23:01:52.820 INFO:tasks.rook.operator.smithi117.stdout:I0308 23:01:52.476940 8 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:52.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.476963 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:52.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.499146 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:01:52.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.510752 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:01:52.916 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:52.928287 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T23:01:53.806 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:53.817800 I | op-mgr: successful modules: prometheus 2022-03-08T23:01:55.034 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:01:55.547 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:01:55.547 INFO:teuthology.orchestra.run.smithi117.stderr:no valid command found; 10 closest matches: 2022-03-08T23:01:55.548 INFO:teuthology.orchestra.run.smithi117.stderr:pg stat 2022-03-08T23:01:55.548 INFO:teuthology.orchestra.run.smithi117.stderr:pg getmap 2022-03-08T23:01:55.548 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:01:55.548 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:01:55.549 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_pools_json 2022-03-08T23:01:55.549 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-pool [...] 2022-03-08T23:01:55.549 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-primary [] [...] 2022-03-08T23:01:55.549 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-osd [] [...] 2022-03-08T23:01:55.549 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls [] [...] 2022-03-08T23:01:55.550 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:01:55.550 INFO:teuthology.orchestra.run.smithi117.stderr:Error EINVAL: invalid command 2022-03-08T23:01:55.551 INFO:teuthology.orchestra.run.smithi117.stderr:command terminated with exit code 22 2022-03-08T23:01:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:01:58.198 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.210124 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T23:01:58.295 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.306968 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:01:58.296 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.306984 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:58.296 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.306992 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:01:58.300 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.312386 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:01:58.547 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.559098 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:58.549 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.559398 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:58.791 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.799774 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:01:58.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.883516 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:01:58.985 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:58.996627 I | op-mon: start running mons 2022-03-08T23:01:59.050 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:59.060280 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:01:59.085 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:59.096753 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:01:59.092 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:59.103942 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:59.092 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:01:59.104087 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:59.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.007415 I | op-mon: targeting the mon count 1 2022-03-08T23:01:59.999 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.010661 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:00.275 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.286598 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:00.275 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.286626 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:00.501 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.512517 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T23:02:00.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.514634 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:00.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.514649 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:00.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.731026 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:00.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.731049 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:00.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.954907 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:00.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:00.954939 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:01.160 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.171491 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:01.160 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.171519 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:02:01.407 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.419056 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:02:01.407 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.419084 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:02:01.477 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.489591 I | op-mon: mon "a" endpoint is [v2:10.106.34.234:3300,v1:10.106.34.234:6789] 2022-03-08T23:02:01.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.506592 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:02:01.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.937034 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:01.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:01.937277 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:02.328 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.340143 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:02:02.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.351453 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:02:02.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.351475 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:02.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.539801 I | op-mon: mons running: [a] 2022-03-08T23:02:02.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.757929 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:02.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.757954 I | op-mon: mons created: 1 2022-03-08T23:02:02.974 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.985852 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:02.978 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:02.990633 I | op-mon: mons running: [a] 2022-03-08T23:02:03.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.212480 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:03.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.212522 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:02:03.436 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.447901 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:02:03.509 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.521816 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T23:02:03.656 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.668068 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:02:03.888 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:03.900205 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:02:04.322 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.333595 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:02:04.322 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.333625 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:02:04.722 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.733809 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:02:04.952 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.963025 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:02:04.953 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.963054 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:02:04.953 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:04.963059 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:02:05.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.258898 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:02:05.278 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.290453 I | op-mgr: start running mgr 2022-03-08T23:02:05.278 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.290488 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:02:05.556 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:02:05.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.758417 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:02:05.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.765741 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:02:05.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.768050 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:02:05.808 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:05.820336 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T23:02:06.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:no valid command found; 10 closest matches: 2022-03-08T23:02:06.010 INFO:teuthology.orchestra.run.smithi117.stderr:pg stat 2022-03-08T23:02:06.010 INFO:teuthology.orchestra.run.smithi117.stderr:pg getmap 2022-03-08T23:02:06.010 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:02:06.010 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:02:06.011 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_pools_json 2022-03-08T23:02:06.011 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-pool [...] 2022-03-08T23:02:06.011 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-primary [] [...] 2022-03-08T23:02:06.011 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls-by-osd [] [...] 2022-03-08T23:02:06.012 INFO:teuthology.orchestra.run.smithi117.stderr:pg ls [] [...] 2022-03-08T23:02:06.012 INFO:teuthology.orchestra.run.smithi117.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:02:06.012 INFO:teuthology.orchestra.run.smithi117.stderr:Error EINVAL: invalid command 2022-03-08T23:02:06.013 INFO:teuthology.orchestra.run.smithi117.stderr:command terminated with exit code 22 2022-03-08T23:02:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:02:06.532 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.544276 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:02:06.533 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.544361 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T23:02:06.533 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.544389 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:02:06.565 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.576876 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:02:06.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.731635 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T23:02:06.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.731661 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:02:06.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:06.731666 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:02:07.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:07.350883 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:02:07.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:07.350915 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:07.766 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:07.777922 I | op-mgr: successful modules: dashboard 2022-03-08T23:02:07.773 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:07.784860 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:02:07.776 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:07.785375 I | op-mgr: successful modules: prometheus 2022-03-08T23:02:08.065 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:08.076782 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T23:02:08.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:08.833260 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T23:02:09.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:09.088752 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T23:02:11.096 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:11.108812 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T23:02:13.494 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:13.506440 I | op-mgr: successful modules: balancer 2022-03-08T23:02:14.168 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:14.180383 I | op-mgr: successful modules: balancer 2022-03-08T23:02:14.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:14.569886 I | cephclient: action set rook backend succeeded on attempt 1 2022-03-08T23:02:14.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:14.569912 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:02:16.017 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status -f json 2022-03-08T23:02:16.464 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:16.476001 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-08T23:02:16.464 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:16.476021 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:02:16.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:16.561 INFO:teuthology.orchestra.run.smithi117.stdout:{"available": true, "backend": "rook"} 2022-03-08T23:02:16.583 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T23:02:16.584 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T23:02:17.149 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch apply osd --all-available-devices 2022-03-08T23:02:17.794 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:17.804977 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:02:17.794 INFO:tasks.rook.operator.smithi117.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:02:17.795 INFO:tasks.rook.operator.smithi117.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:02:17.795 INFO:tasks.rook.operator.smithi117.stdout:   ... // 3 identical fields 2022-03-08T23:02:17.795 INFO:tasks.rook.operator.smithi117.stdout:   Config: nil, 2022-03-08T23:02:17.795 INFO:tasks.rook.operator.smithi117.stdout:   Selection: {}, 2022-03-08T23:02:17.796 INFO:tasks.rook.operator.smithi117.stdout:-  StorageClassDeviceSets: nil, 2022-03-08T23:02:17.796 INFO:tasks.rook.operator.smithi117.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:02:17.796 INFO:tasks.rook.operator.smithi117.stdout:+  { 2022-03-08T23:02:17.796 INFO:tasks.rook.operator.smithi117.stdout:+  Name: "smithi117-lv-1", 2022-03-08T23:02:17.797 INFO:tasks.rook.operator.smithi117.stdout:+  Count: 1, 2022-03-08T23:02:17.797 INFO:tasks.rook.operator.smithi117.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:02:17.797 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:02:17.798 INFO:tasks.rook.operator.smithi117.stdout:+  { 2022-03-08T23:02:17.799 INFO:tasks.rook.operator.smithi117.stdout:+  Name: "smithi117-lv-2", 2022-03-08T23:02:17.799 INFO:tasks.rook.operator.smithi117.stdout:+  Count: 1, 2022-03-08T23:02:17.799 INFO:tasks.rook.operator.smithi117.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:02:17.800 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:02:17.800 INFO:tasks.rook.operator.smithi117.stdout:+  { 2022-03-08T23:02:17.800 INFO:tasks.rook.operator.smithi117.stdout:+  Name: "smithi117-lv-3", 2022-03-08T23:02:17.800 INFO:tasks.rook.operator.smithi117.stdout:+  Count: 1, 2022-03-08T23:02:17.800 INFO:tasks.rook.operator.smithi117.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:02:17.801 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:02:17.801 INFO:tasks.rook.operator.smithi117.stdout:+  { 2022-03-08T23:02:17.801 INFO:tasks.rook.operator.smithi117.stdout:+  Name: "smithi117-lv-4", 2022-03-08T23:02:17.801 INFO:tasks.rook.operator.smithi117.stdout:+  Count: 1, 2022-03-08T23:02:17.802 INFO:tasks.rook.operator.smithi117.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:02:17.802 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:02:17.802 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:02:17.802 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:02:17.802 INFO:tasks.rook.operator.smithi117.stdout:   Annotations: nil, 2022-03-08T23:02:17.803 INFO:tasks.rook.operator.smithi117.stdout:   Labels: nil, 2022-03-08T23:02:17.803 INFO:tasks.rook.operator.smithi117.stdout:   ... // 20 identical fields 2022-03-08T23:02:17.803 INFO:teuthology.orchestra.run.smithi117.stdout:['Success'] 2022-03-08T23:02:17.804 INFO:tasks.rook.operator.smithi117.stdout:  } 2022-03-08T23:02:17.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:17.805083 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:17.817 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-08T23:02:17.818 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:02:17.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:17.830469 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:17.830 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:17.841720 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:02:18.274 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:18.294 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:02:18.294 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:02:23.303 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.315136 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:02:23.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.315159 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:02:23.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.315171 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:02:23.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.319751 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:23.583 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.592724 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:23.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:23.596891 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:24.052 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.064328 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:02:24.191 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.203539 I | op-mon: start running mons 2022-03-08T23:02:24.256 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.267953 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:24.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.316385 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:02:24.314 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.324365 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:24.315 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:24.324908 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:25.139 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:25.150882 I | op-mon: targeting the mon count 1 2022-03-08T23:02:25.155 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:25.166314 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T23:02:26.155 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:26.167083 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:26.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:26.174168 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:26.179 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:26.191268 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:02:26.222 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:26.234147 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:02:26.227 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:26.239674 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:02:28.233 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:28.245351 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:02:28.295 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:02:28.957 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:28.980 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:02:28.980 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:02:33.669 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.680955 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:02:33.669 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.680983 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:02:33.669 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.680998 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:02:33.933 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.945460 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:33.938 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.949988 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:33.941 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:33.950304 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:34.389 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.401826 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:02:34.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.507938 I | op-mon: start running mons 2022-03-08T23:02:34.541 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.553365 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:02:34.573 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.586107 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:02:34.580 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.593191 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:34.584 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:34.593419 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:35.715 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:35.726129 I | op-mon: targeting the mon count 1 2022-03-08T23:02:35.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:35.730258 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:36.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.068480 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:36.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.068521 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:36.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.407484 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:36.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.407510 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:36.734 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.746159 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:36.734 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:36.746201 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:37.101 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.113656 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:37.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.113680 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:37.454 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.466525 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:37.455 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.466553 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:02:37.811 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.823719 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:02:37.812 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.823751 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:02:37.952 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.964372 I | op-mon: mon "a" endpoint is [v2:10.106.34.234:3300,v1:10.106.34.234:6789] 2022-03-08T23:02:37.983 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:37.995747 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:02:37.991 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.003787 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:37.991 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.003989 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:38.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.281463 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:02:38.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.289933 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:02:38.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.289957 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:38.419 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.431516 I | op-mon: mons running: [a] 2022-03-08T23:02:38.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.872754 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:38.861 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:38.872779 I | op-mon: mons created: 1 2022-03-08T23:02:38.981 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:02:39.291 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:39.304156 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:39.299 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:39.312168 I | op-mon: mons running: [a] 2022-03-08T23:02:39.486 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:39.504 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:02:39.505 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:02:39.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:39.731414 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:39.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:39.731461 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:02:40.133 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:40.146045 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:02:40.542 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:40.554041 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:02:40.975 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:40.987288 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:02:41.402 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:41.414781 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:02:41.402 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:41.414802 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:02:41.822 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:41.835085 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:02:42.244 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.256742 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:02:42.245 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.256771 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:02:42.245 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.256777 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:02:42.655 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.668112 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:02:42.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.692107 I | op-mgr: start running mgr 2022-03-08T23:02:42.680 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:42.692133 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:02:43.154 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.166245 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:02:43.164 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.177638 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:02:43.167 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.179904 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:02:43.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.281744 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:02:43.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.281887 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:02:43.287 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.300313 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:02:43.387 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.400096 I | op-osd: creating 1 new PVCs for device set "smithi117-lv-1" 2022-03-08T23:02:43.593 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.606204 I | op-osd: successfully provisioned PVC "smithi117-lv-1-data-07skck" 2022-03-08T23:02:43.594 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.606236 I | op-osd: creating 1 new PVCs for device set "smithi117-lv-2" 2022-03-08T23:02:43.790 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.802945 I | op-osd: successfully provisioned PVC "smithi117-lv-2-data-0zc9vd" 2022-03-08T23:02:43.791 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:43.803104 I | op-osd: creating 1 new PVCs for device set "smithi117-lv-3" 2022-03-08T23:02:43.991 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.003962 I | op-osd: successfully provisioned PVC "smithi117-lv-3-data-06h6fm" 2022-03-08T23:02:43.992 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.004011 I | op-osd: creating 1 new PVCs for device set "smithi117-lv-4" 2022-03-08T23:02:44.040 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.052763 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:02:44.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.057483 I | op-mgr: successful modules: dashboard 2022-03-08T23:02:44.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.061906 I | op-mgr: successful modules: prometheus 2022-03-08T23:02:44.189 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.201957 I | op-osd: successfully provisioned PVC "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:02:44.206 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.219093 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:02:44.567 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.579482 I | op-mgr: successful modules: balancer 2022-03-08T23:02:44.628 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.640959 I | op-osd: started OSD provisioning job for PVC "smithi117-lv-1-data-07skck" 2022-03-08T23:02:44.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:44.640990 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-0zc9vd" 2022-03-08T23:02:45.066 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.079189 I | op-osd: started OSD provisioning job for PVC "smithi117-lv-2-data-0zc9vd" 2022-03-08T23:02:45.067 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.079211 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:02:45.403 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.415719 I | op-osd: started OSD provisioning job for PVC "smithi117-lv-3-data-06h6fm" 2022-03-08T23:02:45.403 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.415744 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:02:45.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.748119 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:02:45.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.815165 I | op-osd: started OSD provisioning job for PVC "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:02:45.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.815197 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:02:45.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:45.815207 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:02:45.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:46.006473 I | op-osd: OSD orchestration status for PVC smithi117-lv-1-data-07skck is "starting" 2022-03-08T23:02:45.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:46.006506 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-0zc9vd is "starting" 2022-03-08T23:02:45.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:46.006520 I | op-osd: OSD orchestration status for PVC smithi117-lv-3-data-06h6fm is "starting" 2022-03-08T23:02:45.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:02:46.006532 I | op-osd: OSD orchestration status for PVC smithi117-lv-4-data-0lbbtm is "starting" 2022-03-08T23:02:49.506 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:02:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:02:50.011 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:02:50.012 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:00.013 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:00.562 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:00.584 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:03:00.585 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:10.586 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:11.438 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:11.451411 I | op-osd: OSD orchestration status for node smithi117-lv-2-data-0zc9vd is "orchestrating" 2022-03-08T23:03:12.103 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:12.116759 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-0zc9vd is "orchestrating" 2022-03-08T23:03:12.171 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:12.188 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:03:12.189 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:15.425 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:15.438540 I | op-osd: OSD orchestration status for node smithi117-lv-3-data-06h6fm is "orchestrating" 2022-03-08T23:03:15.728 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:15.741738 I | op-osd: OSD orchestration status for node smithi117-lv-4-data-0lbbtm is "orchestrating" 2022-03-08T23:03:15.862 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:15.875425 I | op-osd: OSD orchestration status for PVC smithi117-lv-3-data-06h6fm is "orchestrating" 2022-03-08T23:03:16.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:16.506602 I | op-osd: OSD orchestration status for PVC smithi117-lv-4-data-0lbbtm is "orchestrating" 2022-03-08T23:03:18.950 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:18.963806 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-0zc9vd is "completed" 2022-03-08T23:03:18.951 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:18.963850 I | op-osd: creating OSD 0 on PVC "smithi117-lv-2-data-0zc9vd" 2022-03-08T23:03:18.951 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:18.963866 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-0zc9vd" 2022-03-08T23:03:19.914 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:19.927905 I | op-osd: OSD orchestration status for node smithi117-lv-1-data-07skck is "orchestrating" 2022-03-08T23:03:20.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:20.352450 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:20.469 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:20.482331 I | op-osd: OSD orchestration status for PVC smithi117-lv-1-data-07skck is "orchestrating" 2022-03-08T23:03:20.521 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:20.534036 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:22.190 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:22.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:22.516081 I | op-osd: OSD orchestration status for PVC smithi117-lv-3-data-06h6fm is "completed" 2022-03-08T23:03:22.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:22.516155 I | op-osd: creating OSD 1 on PVC "smithi117-lv-3-data-06h6fm" 2022-03-08T23:03:22.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:22.516171 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:03:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:23.573 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:03:23.574 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:23.576 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:23.587063 I | op-osd: OSD orchestration status for PVC smithi117-lv-4-data-0lbbtm is "completed" 2022-03-08T23:03:23.576 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:23.587141 I | op-osd: creating OSD 2 on PVC "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:03:23.576 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:23.587175 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:03:28.714 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:28.728047 I | op-osd: OSD orchestration status for PVC smithi117-lv-1-data-07skck is "completed" 2022-03-08T23:03:28.715 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:28.728126 I | op-osd: creating OSD 3 on PVC "smithi117-lv-1-data-07skck" 2022-03-08T23:03:28.715 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:28.728159 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:03:30.762 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:30.775989 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:03:30.763 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:03:30.776007 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:03:33.575 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:34.829 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:34.847 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:03:34.847 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:44.849 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:46.319 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:46.337 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:03:46.338 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:03:56.339 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:03:57.298 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:03:57.322 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:03:57.323 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T23:04:07.324 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:04:07.875 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:07.894 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":18,"num_osds":4,"num_up_osds":3,"osd_up_since":1646780646,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:04:07.895 INFO:tasks.rook: have 3/4 OSDs 2022-03-08T23:04:17.897 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd stat -f json 2022-03-08T23:04:18.417 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:18.436 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"num_osds":4,"num_up_osds":4,"osd_up_since":1646780648,"num_in_osds":4,"osd_in_since":1646780602,"num_remapped_pgs":0} 2022-03-08T23:04:18.437 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T23:04:18.437 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- cat /etc/ceph/ceph.conf 2022-03-08T23:04:18.625 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2022-03-08T23:04:18.625 INFO:teuthology.orchestra.run.smithi117.stdout:mon_host = 10.106.34.234:6789 2022-03-08T23:04:18.626 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:18.626 INFO:teuthology.orchestra.run.smithi117.stdout:[client.admin] 2022-03-08T23:04:18.626 INFO:teuthology.orchestra.run.smithi117.stdout:keyring = /etc/ceph/keyring 2022-03-08T23:04:18.630 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- cat /etc/ceph/keyring 2022-03-08T23:04:18.828 INFO:teuthology.orchestra.run.smithi117.stdout:[client.admin] 2022-03-08T23:04:18.829 INFO:teuthology.orchestra.run.smithi117.stdout:key = AQAf4CditNKLIhAAM7Y9DxfQ0m3ATcu5DasSsA== 2022-03-08T23:04:18.833 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2022-03-08T23:04:18.846 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:04:18.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:04:18.860 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:04:18.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:04:18.922 INFO:tasks.rook:Setting up client nodes... 2022-03-08T23:04:18.923 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:04:19.530 INFO:teuthology.orchestra.run.smithi117.stdout:[client.a] 2022-03-08T23:04:19.531 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDz4CdiA3l0HRAAbswpqRmrZ5NXWvdiQCxrIg== 2022-03-08T23:04:19.552 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:04:19.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T23:04:19.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T23:04:19.574 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd dump --format=json 2022-03-08T23:04:20.138 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:20.138 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"fsid":"7b2ed255-7ddb-41a4-9a7b-7bd1bb346968","created":"2022-03-08T23:01:33.503478+0000","modified":"2022-03-08T23:04:11.038881+0000","last_up_change":"2022-03-08T23:04:08.794011+0000","last_in_change":"2022-03-08T23:03:22.156057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:04:08.580440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cafa5ff4-7e33-4ca7-9d3c-91518ecd4953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6800","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6801","nonce":139206827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6802","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6803","nonce":139206827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6806","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6807","nonce":139206827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6804","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6805","nonce":139206827}]},"public_addr":"10.251.165.150:6801/139206827","cluster_addr":"10.251.165.150:6803/139206827","heartbeat_back_addr":"10.251.165.150:6807/139206827","heartbeat_front_addr":"10.251.165.150:6805/139206827","state":["exists","up"]},{"osd":1,"uuid":"3f89676d-83f8-4788-b56b-8c4fadedf82e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6800","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6801","nonce":1798833918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6802","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6803","nonce":1798833918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6806","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6807","nonce":1798833918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6804","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6805","nonce":1798833918}]},"public_addr":"10.251.165.152:6801/1798833918","cluster_addr":"10.251.165.152:6803/1798833918","heartbeat_back_addr":"10.251.165.152:6807/1798833918","heartbeat_front_addr":"10.251.165.152:6805/1798833918","state":["exists","up"]},{"osd":2,"uuid":"8c140f72-992a-4e15-9e1b-ce0b477ddaf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6800","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6801","nonce":3176374828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6802","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6803","nonce":3176374828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6806","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6807","nonce":3176374828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6804","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6805","nonce":3176374828}]},"public_addr":"10.251.165.153:6801/3176374828","cluster_addr":"10.251.165.153:6803/3176374828","heartbeat_back_addr":"10.251.165.153:6807/3176374828","heartbeat_front_addr":"10.251.165.153:6805/3176374828","state":["exists","up"]},{"osd":3,"uuid":"e70f85f7-e43d-41ab-9dca-ef95ba4c34d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6800","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6801","nonce":4184010390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6802","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6803","nonce":4184010390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6806","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6807","nonce":4184010390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6804","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6805","nonce":4184010390}]},"public_addr":"10.251.165.154:6801/4184010390","cluster_addr":"10.251.165.154:6803/4184010390","heartbeat_back_addr":"10.251.165.154:6807/4184010390","heartbeat_front_addr":"10.251.165.154:6805/4184010390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:03:59.914699+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:02.172012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:05.247038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:07.266949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:04:20.156 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T23:04:08.580440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T23:04:20.156 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd pool get .mgr pg_num 2022-03-08T23:04:20.665 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2022-03-08T23:04:20.693 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T23:04:20.695 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T23:04:20.696 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph mgr dump --format=json 2022-03-08T23:04:21.263 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:21.282 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"active_gid":4199,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.142:6800","nonce":2866588532},{"type":"v1","addr":"10.251.165.142:6801","nonce":2866588532}]},"active_addr":"10.251.165.142:6801/2866588532","active_change":"2022-03-08T23:02:11.912419+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.251.165.142:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.251.165.142:0","nonce":65507096}]},{"addrvec":[{"type":"v2","addr":"10.251.165.142:0","nonce":1735178}]},{"addrvec":[{"type":"v2","addr":"10.251.165.142:0","nonce":2442884525}]},{"addrvec":[{"type":"v2","addr":"10.251.165.142:0","nonce":2909344673}]}]}} 2022-03-08T23:04:21.285 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T23:04:21.286 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T23:04:21.286 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd dump --format=json 2022-03-08T23:04:21.814 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:21.815 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"fsid":"7b2ed255-7ddb-41a4-9a7b-7bd1bb346968","created":"2022-03-08T23:01:33.503478+0000","modified":"2022-03-08T23:04:11.038881+0000","last_up_change":"2022-03-08T23:04:08.794011+0000","last_in_change":"2022-03-08T23:03:22.156057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:04:08.580440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cafa5ff4-7e33-4ca7-9d3c-91518ecd4953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6800","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6801","nonce":139206827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6802","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6803","nonce":139206827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6806","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6807","nonce":139206827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6804","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6805","nonce":139206827}]},"public_addr":"10.251.165.150:6801/139206827","cluster_addr":"10.251.165.150:6803/139206827","heartbeat_back_addr":"10.251.165.150:6807/139206827","heartbeat_front_addr":"10.251.165.150:6805/139206827","state":["exists","up"]},{"osd":1,"uuid":"3f89676d-83f8-4788-b56b-8c4fadedf82e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6800","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6801","nonce":1798833918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6802","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6803","nonce":1798833918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6806","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6807","nonce":1798833918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6804","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6805","nonce":1798833918}]},"public_addr":"10.251.165.152:6801/1798833918","cluster_addr":"10.251.165.152:6803/1798833918","heartbeat_back_addr":"10.251.165.152:6807/1798833918","heartbeat_front_addr":"10.251.165.152:6805/1798833918","state":["exists","up"]},{"osd":2,"uuid":"8c140f72-992a-4e15-9e1b-ce0b477ddaf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6800","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6801","nonce":3176374828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6802","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6803","nonce":3176374828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6806","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6807","nonce":3176374828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6804","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6805","nonce":3176374828}]},"public_addr":"10.251.165.153:6801/3176374828","cluster_addr":"10.251.165.153:6803/3176374828","heartbeat_back_addr":"10.251.165.153:6807/3176374828","heartbeat_front_addr":"10.251.165.153:6805/3176374828","state":["exists","up"]},{"osd":3,"uuid":"e70f85f7-e43d-41ab-9dca-ef95ba4c34d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6800","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6801","nonce":4184010390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6802","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6803","nonce":4184010390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6806","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6807","nonce":4184010390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6804","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6805","nonce":4184010390}]},"public_addr":"10.251.165.154:6801/4184010390","cluster_addr":"10.251.165.154:6803/4184010390","heartbeat_back_addr":"10.251.165.154:6807/4184010390","heartbeat_front_addr":"10.251.165.154:6805/4184010390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:03:59.914699+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:02.172012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:05.247038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:07.266949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:04:21.837 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T23:04:21.837 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd dump --format=json 2022-03-08T23:04:22.397 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:22.397 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"fsid":"7b2ed255-7ddb-41a4-9a7b-7bd1bb346968","created":"2022-03-08T23:01:33.503478+0000","modified":"2022-03-08T23:04:11.038881+0000","last_up_change":"2022-03-08T23:04:08.794011+0000","last_in_change":"2022-03-08T23:03:22.156057+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:04:08.580440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cafa5ff4-7e33-4ca7-9d3c-91518ecd4953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6800","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6801","nonce":139206827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6802","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6803","nonce":139206827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6806","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6807","nonce":139206827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.150:6804","nonce":139206827},{"type":"v1","addr":"10.251.165.150:6805","nonce":139206827}]},"public_addr":"10.251.165.150:6801/139206827","cluster_addr":"10.251.165.150:6803/139206827","heartbeat_back_addr":"10.251.165.150:6807/139206827","heartbeat_front_addr":"10.251.165.150:6805/139206827","state":["exists","up"]},{"osd":1,"uuid":"3f89676d-83f8-4788-b56b-8c4fadedf82e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6800","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6801","nonce":1798833918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6802","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6803","nonce":1798833918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6806","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6807","nonce":1798833918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.152:6804","nonce":1798833918},{"type":"v1","addr":"10.251.165.152:6805","nonce":1798833918}]},"public_addr":"10.251.165.152:6801/1798833918","cluster_addr":"10.251.165.152:6803/1798833918","heartbeat_back_addr":"10.251.165.152:6807/1798833918","heartbeat_front_addr":"10.251.165.152:6805/1798833918","state":["exists","up"]},{"osd":2,"uuid":"8c140f72-992a-4e15-9e1b-ce0b477ddaf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6800","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6801","nonce":3176374828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6802","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6803","nonce":3176374828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6806","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6807","nonce":3176374828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.153:6804","nonce":3176374828},{"type":"v1","addr":"10.251.165.153:6805","nonce":3176374828}]},"public_addr":"10.251.165.153:6801/3176374828","cluster_addr":"10.251.165.153:6803/3176374828","heartbeat_back_addr":"10.251.165.153:6807/3176374828","heartbeat_front_addr":"10.251.165.153:6805/3176374828","state":["exists","up"]},{"osd":3,"uuid":"e70f85f7-e43d-41ab-9dca-ef95ba4c34d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6800","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6801","nonce":4184010390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6802","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6803","nonce":4184010390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6806","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6807","nonce":4184010390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.165.154:6804","nonce":4184010390},{"type":"v1","addr":"10.251.165.154:6805","nonce":4184010390}]},"public_addr":"10.251.165.154:6801/4184010390","cluster_addr":"10.251.165.154:6803/4184010390","heartbeat_back_addr":"10.251.165.154:6807/4184010390","heartbeat_front_addr":"10.251.165.154:6805/4184010390","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:03:59.914699+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:02.172012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:05.247038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:04:07.266949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:04:22.418 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph tell osd.0 flush_pg_stats 2022-03-08T23:04:22.419 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph tell osd.1 flush_pg_stats 2022-03-08T23:04:22.420 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph tell osd.2 flush_pg_stats 2022-03-08T23:04:22.420 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph tell osd.3 flush_pg_stats 2022-03-08T23:04:22.788 INFO:teuthology.orchestra.run.smithi117.stdout:73014444038 2022-03-08T23:04:22.789 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.2 2022-03-08T23:04:22.861 INFO:teuthology.orchestra.run.smithi117.stdout:81604378629 2022-03-08T23:04:22.862 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.3 2022-03-08T23:04:22.882 INFO:teuthology.orchestra.run.smithi117.stdout:55834574853 2022-03-08T23:04:22.882 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.0 2022-03-08T23:04:23.024 INFO:teuthology.orchestra.run.smithi117.stdout:64424509445 2022-03-08T23:04:23.025 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.1 2022-03-08T23:04:23.371 INFO:teuthology.orchestra.run.smithi117.stdout:81604378628 2022-03-08T23:04:23.384 INFO:teuthology.orchestra.run.smithi117.stdout:73014444036 2022-03-08T23:04:23.396 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378629 got 81604378628 for osd.3 2022-03-08T23:04:23.404 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444036 for osd.2 2022-03-08T23:04:23.439 INFO:teuthology.orchestra.run.smithi117.stdout:55834574852 2022-03-08T23:04:23.458 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2022-03-08T23:04:23.584 INFO:teuthology.orchestra.run.smithi117.stdout:64424509444 2022-03-08T23:04:23.603 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.1 2022-03-08T23:04:24.397 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.3 2022-03-08T23:04:24.406 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.2 2022-03-08T23:04:24.460 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.0 2022-03-08T23:04:24.604 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.1 2022-03-08T23:04:25.027 INFO:teuthology.orchestra.run.smithi117.stdout:73014444036 2022-03-08T23:04:25.047 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444036 for osd.2 2022-03-08T23:04:25.152 INFO:teuthology.orchestra.run.smithi117.stdout:81604378629 2022-03-08T23:04:25.153 INFO:teuthology.orchestra.run.smithi117.stdout:64424509445 2022-03-08T23:04:25.170 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378629 got 81604378629 for osd.3 2022-03-08T23:04:25.171 DEBUG:teuthology.parallel:result is None 2022-03-08T23:04:25.173 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.1 2022-03-08T23:04:25.173 DEBUG:teuthology.parallel:result is None 2022-03-08T23:04:25.249 INFO:teuthology.orchestra.run.smithi117.stdout:55834574854 2022-03-08T23:04:25.274 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2022-03-08T23:04:25.274 DEBUG:teuthology.parallel:result is None 2022-03-08T23:04:26.049 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd last-stat-seq osd.2 2022-03-08T23:04:26.549 INFO:teuthology.orchestra.run.smithi117.stdout:73014444038 2022-03-08T23:04:26.572 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.2 2022-03-08T23:04:26.572 DEBUG:teuthology.parallel:result is None 2022-03-08T23:04:26.573 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T23:04:26.573 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph pg dump --format=json 2022-03-08T23:04:27.118 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:27.123 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2022-03-08T23:04:27.140 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-08T23:04:26.076561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23684,"kb_used_data":2084,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":374956924,"statfs":{"total":383980142592,"available":383955890176,"internally_reserved":0,"allocated":2134016,"data_stored":1579460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":38,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003392"},"pg_stats":[{"pgid":"1.0","version":"21'97","reported_seq":137,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-03-08T23:04:14.288385+0000","last_change":"2022-03-08T23:04:10.322145+0000","last_active":"2022-03-08T23:04:14.288385+0000","last_peered":"2022-03-08T23:04:14.288385+0000","last_clean":"2022-03-08T23:04:14.288385+0000","last_became_active":"2022-03-08T23:04:10.321543+0000","last_became_peered":"2022-03-08T23:04:10.321543+0000","last_unstale":"2022-03-08T23:04:14.288385+0000","last_undegraded":"2022-03-08T23:04:14.288385+0000","last_fullsized":"2022-03-08T23:04:14.288385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:04:08.794011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:04:08.794011+0000","last_clean_scrub_stamp":"2022-03-08T23:04:08.794011+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T03:27:18.662467+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4984,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740168,"statfs":{"total":95995035648,"available":95989932032,"internally_reserved":0,"allocated":180224,"data_stored":44648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.71}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:04:27.141 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph pg dump --format=json 2022-03-08T23:04:27.651 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2022-03-08T23:04:27.669 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-08T23:04:26.076561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23684,"kb_used_data":2084,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":374956924,"statfs":{"total":383980142592,"available":383955890176,"internally_reserved":0,"allocated":2134016,"data_stored":1579460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":38,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003392"},"pg_stats":[{"pgid":"1.0","version":"21'97","reported_seq":137,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-03-08T23:04:14.288385+0000","last_change":"2022-03-08T23:04:10.322145+0000","last_active":"2022-03-08T23:04:14.288385+0000","last_peered":"2022-03-08T23:04:14.288385+0000","last_clean":"2022-03-08T23:04:14.288385+0000","last_became_active":"2022-03-08T23:04:10.321543+0000","last_became_peered":"2022-03-08T23:04:10.321543+0000","last_unstale":"2022-03-08T23:04:14.288385+0000","last_undegraded":"2022-03-08T23:04:14.288385+0000","last_fullsized":"2022-03-08T23:04:14.288385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:04:08.794011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:04:08.794011+0000","last_clean_scrub_stamp":"2022-03-08T23:04:08.794011+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T03:27:18.662467+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4984,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740168,"statfs":{"total":95995035648,"available":95989932032,"internally_reserved":0,"allocated":180224,"data_stored":44648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.71}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":651264,"data_stored":511604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:04:27.670 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T23:04:27.671 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T23:04:27.671 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T23:04:27.671 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph health --format=json 2022-03-08T23:04:28.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:28.190 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T23:04:28.207 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T23:04:28.208 INFO:tasks.rook:Rook complete, yielding 2022-03-08T23:04:28.208 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:04:28.222 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph -s 2022-03-08T23:04:28.782 INFO:teuthology.orchestra.run.smithi117.stdout: cluster: 2022-03-08T23:04:28.783 INFO:teuthology.orchestra.run.smithi117.stdout: id: 7b2ed255-7ddb-41a4-9a7b-7bd1bb346968 2022-03-08T23:04:28.783 INFO:teuthology.orchestra.run.smithi117.stdout: health: HEALTH_OK 2022-03-08T23:04:28.783 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:28.783 INFO:teuthology.orchestra.run.smithi117.stdout: services: 2022-03-08T23:04:28.784 INFO:teuthology.orchestra.run.smithi117.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-08T23:04:28.784 INFO:teuthology.orchestra.run.smithi117.stdout: mgr: a(active, since 2m) 2022-03-08T23:04:28.784 INFO:teuthology.orchestra.run.smithi117.stdout: osd: 4 osds: 4 up (since 20s), 4 in (since 66s) 2022-03-08T23:04:28.784 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:28.784 INFO:teuthology.orchestra.run.smithi117.stdout: data: 2022-03-08T23:04:28.785 INFO:teuthology.orchestra.run.smithi117.stdout: pools: 1 pools, 1 pgs 2022-03-08T23:04:28.785 INFO:teuthology.orchestra.run.smithi117.stdout: objects: 2 objects, 577 KiB 2022-03-08T23:04:28.785 INFO:teuthology.orchestra.run.smithi117.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-03-08T23:04:28.785 INFO:teuthology.orchestra.run.smithi117.stdout: pgs: 1 active+clean 2022-03-08T23:04:28.786 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:04:28.804 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status 2022-03-08T23:04:29.423 INFO:teuthology.orchestra.run.smithi117.stdout:Backend: rook 2022-03-08T23:04:29.424 INFO:teuthology.orchestra.run.smithi117.stdout:Available: Yes 2022-03-08T23:04:29.447 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ps 2022-03-08T23:04:29.954 INFO:teuthology.orchestra.run.smithi117.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:04:29.954 INFO:teuthology.orchestra.run.smithi117.stdout:crashcollector.crash smithi117 running (69s) 0s ago 69s - - 4b871cde5e1a 2022-03-08T23:04:29.954 INFO:teuthology.orchestra.run.smithi117.stdout:mgr.a smithi117 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T23:04:29.955 INFO:teuthology.orchestra.run.smithi117.stdout:mon.a smithi117 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:04:29.955 INFO:teuthology.orchestra.run.smithi117.stdout:osd.0 smithi117 running (70s) 0s ago 70s - - 4b871cde5e1a 2022-03-08T23:04:29.955 INFO:teuthology.orchestra.run.smithi117.stdout:osd.1 smithi117 running (65s) 0s ago 66s - - 4b871cde5e1a 2022-03-08T23:04:29.955 INFO:teuthology.orchestra.run.smithi117.stdout:osd.2 smithi117 running (64s) 0s ago 64s - - 4b871cde5e1a 2022-03-08T23:04:29.956 INFO:teuthology.orchestra.run.smithi117.stdout:osd.3 smithi117 running (59s) 0s ago 59s - - 4b871cde5e1a 2022-03-08T23:04:29.975 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls 2022-03-08T23:04:30.583 INFO:teuthology.orchestra.run.smithi117.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:04:30.583 INFO:teuthology.orchestra.run.smithi117.stdout:crash 1/1 0s ago 70s * 2022-03-08T23:04:30.584 INFO:teuthology.orchestra.run.smithi117.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T23:04:30.584 INFO:teuthology.orchestra.run.smithi117.stdout:mon 1/1 0s ago 3m count:1 2022-03-08T23:04:30.584 INFO:teuthology.orchestra.run.smithi117.stdout:osd 8 0s ago 71s 2022-03-08T23:04:30.584 INFO:teuthology.orchestra.run.smithi117.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:04:30.605 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch device ls 2022-03-08T23:04:31.154 INFO:teuthology.orchestra.run.smithi117.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:04:31.155 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:04:31.155 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:04:31.155 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:04:31.156 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:04:31.178 INFO:teuthology.run_tasks:Running task install... 2022-03-08T23:04:31.309 DEBUG:teuthology.task.install:project ceph 2022-03-08T23:04:31.338 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'}} 2022-03-08T23:04:31.339 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'} 2022-03-08T23:04:31.339 INFO:teuthology.task.install:Using flavor: default 2022-03-08T23:04:31.346 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-08T23:04:31.346 INFO:teuthology.task.install:extra packages: [] 2022-03-08T23:04:31.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-03-08T23:04:31.397 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-03-08T23:04:31.402 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:04:31.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-key list | grep Ceph 2022-03-08T23:04:31.501 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-08T23:04:31.556 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-08T23:04:31.556 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph.com (release key) 2022-03-08T23:04:31.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-03-08T23:04:31.606 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-03-08T23:04:31.612 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:04:31.612 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-08T23:04:31.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T23:04:31.786 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default/ 2022-03-08T23:04:31.786 INFO:teuthology.task.install.deb:Package version is 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T23:04:31.902 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:04:31.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-08T23:04:31.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2022-03-08T23:04:32.215 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:04:32.216 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:04:32.259 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-03-08T23:04:32.286 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:04:32.339 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal InRelease 2022-03-08T23:04:32.362 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-08T23:04:32.368 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:04:32.387 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:8 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release.gpg 2022-03-08T23:04:32.430 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2022-03-08T23:04:32.505 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main i386 Packages [3,360 B] 2022-03-08T23:04:32.908 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 28.3 kB in 1s (34.9 kB/s) 2022-03-08T23:04:33.845 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T23:04:33.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11025-g7a44e9eb-1focal cephadm=17.0.0-11025-g7a44e9eb-1focal ceph-mds=17.0.0-11025-g7a44e9eb-1focal ceph-mgr=17.0.0-11025-g7a44e9eb-1focal ceph-common=17.0.0-11025-g7a44e9eb-1focal ceph-fuse=17.0.0-11025-g7a44e9eb-1focal ceph-test=17.0.0-11025-g7a44e9eb-1focal ceph-volume=17.0.0-11025-g7a44e9eb-1focal radosgw=17.0.0-11025-g7a44e9eb-1focal python3-rados=17.0.0-11025-g7a44e9eb-1focal python3-rgw=17.0.0-11025-g7a44e9eb-1focal python3-cephfs=17.0.0-11025-g7a44e9eb-1focal python3-rbd=17.0.0-11025-g7a44e9eb-1focal libcephfs2=17.0.0-11025-g7a44e9eb-1focal libcephfs-dev=17.0.0-11025-g7a44e9eb-1focal librados2=17.0.0-11025-g7a44e9eb-1focal librbd1=17.0.0-11025-g7a44e9eb-1focal rbd-fuse=17.0.0-11025-g7a44e9eb-1focal 2022-03-08T23:04:33.942 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-03-08T23:04:34.126 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-03-08T23:04:34.126 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-03-08T23:04:34.287 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2022-03-08T23:04:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-08T23:04:34.288 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-08T23:04:34.289 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2022-03-08T23:04:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-08T23:04:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-08T23:04:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T23:04:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T23:04:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T23:04:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T23:04:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-08T23:04:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T23:04:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T23:04:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T23:04:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T23:04:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T23:04:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T23:04:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-08T23:04:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-08T23:04:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-08T23:04:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-08T23:04:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-08T23:04:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-08T23:04:34.294 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2022-03-08T23:04:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-08T23:04:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-08T23:04:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-08T23:04:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-08T23:04:34.296 INFO:teuthology.orchestra.run.smithi117.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-08T23:04:34.296 INFO:teuthology.orchestra.run.smithi117.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-08T23:04:34.296 INFO:teuthology.orchestra.run.smithi117.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-08T23:04:34.296 INFO:teuthology.orchestra.run.smithi117.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-08T23:04:34.297 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2022-03-08T23:04:34.297 INFO:teuthology.orchestra.run.smithi117.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-08T23:04:34.399 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-03-08T23:04:34.400 INFO:teuthology.orchestra.run.smithi117.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-08T23:04:34.400 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-08T23:04:34.402 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-08T23:04:34.402 INFO:teuthology.orchestra.run.smithi117.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T23:04:34.402 INFO:teuthology.orchestra.run.smithi117.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T23:04:34.402 INFO:teuthology.orchestra.run.smithi117.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T23:04:34.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T23:04:34.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-08T23:04:34.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T23:04:34.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T23:04:34.404 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T23:04:34.404 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T23:04:34.404 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T23:04:34.404 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T23:04:34.404 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-08T23:04:34.405 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-08T23:04:34.405 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-08T23:04:34.405 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-08T23:04:34.405 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-08T23:04:34.406 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-08T23:04:34.406 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-08T23:04:34.406 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages will be upgraded: 2022-03-08T23:04:34.407 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 librbd1 2022-03-08T23:04:34.639 INFO:teuthology.orchestra.run.smithi117.stdout:2 upgraded, 95 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:04:34.640 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 181 MB of archives. 2022-03-08T23:04:34.640 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 845 MB of additional disk space will be used. 2022-03-08T23:04:34.640 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11025-g7a44e9eb-1focal [3,484 kB] 2022-03-08T23:04:34.648 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-08T23:04:35.021 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-08T23:04:35.026 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-08T23:04:35.038 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-08T23:04:35.040 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-08T23:04:35.145 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11025-g7a44e9eb-1focal [3,533 kB] 2022-03-08T23:04:35.208 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-08T23:04:35.216 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-08T23:04:35.250 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-08T23:04:35.256 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-08T23:04:35.261 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-08T23:04:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-08T23:04:35.283 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-08T23:04:35.284 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-08T23:04:35.285 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-08T23:04:35.291 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-08T23:04:35.374 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-08T23:04:35.375 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-08T23:04:35.381 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-08T23:04:35.382 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-08T23:04:35.386 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-08T23:04:35.388 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-08T23:04:35.389 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-08T23:04:35.401 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-08T23:04:35.402 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-08T23:04:35.455 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-08T23:04:35.473 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11025-g7a44e9eb-1focal [755 kB] 2022-03-08T23:04:35.485 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-08T23:04:35.488 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-08T23:04:35.494 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-08T23:04:35.495 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-08T23:04:35.497 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-08T23:04:35.500 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-08T23:04:35.501 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-08T23:04:35.502 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-08T23:04:35.525 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11025-g7a44e9eb-1focal [412 kB] 2022-03-08T23:04:35.538 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-08T23:04:35.539 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11025-g7a44e9eb-1focal [22.4 kB] 2022-03-08T23:04:35.575 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11025-g7a44e9eb-1focal [200 kB] 2022-03-08T23:04:35.585 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11025-g7a44e9eb-1focal [41.9 kB] 2022-03-08T23:04:35.602 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11025-g7a44e9eb-1focal [385 kB] 2022-03-08T23:04:35.618 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11025-g7a44e9eb-1focal [4,162 kB] 2022-03-08T23:04:35.844 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11025-g7a44e9eb-1focal [116 kB] 2022-03-08T23:04:35.914 INFO:teuthology.orchestra.run.smithi117.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11025-g7a44e9eb-1focal [478 kB] 2022-03-08T23:04:36.031 INFO:teuthology.orchestra.run.smithi117.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-08T23:04:36.135 INFO:teuthology.orchestra.run.smithi117.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-08T23:04:36.137 INFO:teuthology.orchestra.run.smithi117.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-08T23:04:36.157 INFO:teuthology.orchestra.run.smithi117.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11025-g7a44e9eb-1focal [24.4 MB] 2022-03-08T23:04:36.173 INFO:teuthology.orchestra.run.smithi117.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-08T23:04:36.173 INFO:teuthology.orchestra.run.smithi117.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-08T23:04:36.174 INFO:teuthology.orchestra.run.smithi117.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-08T23:04:36.174 INFO:teuthology.orchestra.run.smithi117.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-08T23:04:36.175 INFO:teuthology.orchestra.run.smithi117.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-08T23:04:36.175 INFO:teuthology.orchestra.run.smithi117.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-08T23:04:36.176 INFO:teuthology.orchestra.run.smithi117.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-08T23:04:36.241 INFO:teuthology.orchestra.run.smithi117.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-08T23:04:36.326 INFO:teuthology.orchestra.run.smithi117.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-08T23:04:36.326 INFO:teuthology.orchestra.run.smithi117.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-08T23:04:36.332 INFO:teuthology.orchestra.run.smithi117.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-08T23:04:36.334 INFO:teuthology.orchestra.run.smithi117.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-08T23:04:36.338 INFO:teuthology.orchestra.run.smithi117.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-08T23:04:36.338 INFO:teuthology.orchestra.run.smithi117.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-08T23:04:36.340 INFO:teuthology.orchestra.run.smithi117.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-08T23:04:36.347 INFO:teuthology.orchestra.run.smithi117.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-08T23:04:36.362 INFO:teuthology.orchestra.run.smithi117.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-08T23:04:36.454 INFO:teuthology.orchestra.run.smithi117.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-08T23:04:36.454 INFO:teuthology.orchestra.run.smithi117.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-08T23:04:36.456 INFO:teuthology.orchestra.run.smithi117.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-08T23:04:36.458 INFO:teuthology.orchestra.run.smithi117.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-08T23:04:36.492 INFO:teuthology.orchestra.run.smithi117.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-08T23:04:36.492 INFO:teuthology.orchestra.run.smithi117.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-08T23:04:36.493 INFO:teuthology.orchestra.run.smithi117.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-08T23:04:36.493 INFO:teuthology.orchestra.run.smithi117.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-08T23:04:36.494 INFO:teuthology.orchestra.run.smithi117.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-08T23:04:36.552 INFO:teuthology.orchestra.run.smithi117.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-08T23:04:36.553 INFO:teuthology.orchestra.run.smithi117.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-08T23:04:36.553 INFO:teuthology.orchestra.run.smithi117.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-08T23:04:36.557 INFO:teuthology.orchestra.run.smithi117.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-08T23:04:37.304 INFO:teuthology.orchestra.run.smithi117.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11025-g7a44e9eb-1focal [6,433 kB] 2022-03-08T23:04:37.475 INFO:teuthology.orchestra.run.smithi117.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11025-g7a44e9eb-1focal [211 kB] 2022-03-08T23:04:37.479 INFO:teuthology.orchestra.run.smithi117.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11025-g7a44e9eb-1focal [151 kB] 2022-03-08T23:04:37.494 INFO:teuthology.orchestra.run.smithi117.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11025-g7a44e9eb-1focal [1,574 kB] 2022-03-08T23:04:37.589 INFO:teuthology.orchestra.run.smithi117.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11025-g7a44e9eb-1focal [7,824 kB] 2022-03-08T23:04:37.837 INFO:teuthology.orchestra.run.smithi117.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11025-g7a44e9eb-1focal [28.6 MB] 2022-03-08T23:04:39.144 INFO:teuthology.orchestra.run.smithi117.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11025-g7a44e9eb-1focal [3,808 B] 2022-03-08T23:04:39.218 INFO:teuthology.orchestra.run.smithi117.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [863 kB] 2022-03-08T23:04:39.223 INFO:teuthology.orchestra.run.smithi117.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11025-g7a44e9eb-1focal [2,396 kB] 2022-03-08T23:04:39.485 INFO:teuthology.orchestra.run.smithi117.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11025-g7a44e9eb-1focal [1,677 kB] 2022-03-08T23:04:39.555 INFO:teuthology.orchestra.run.smithi117.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11025-g7a44e9eb-1focal [7,889 kB] 2022-03-08T23:04:39.890 INFO:teuthology.orchestra.run.smithi117.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11025-g7a44e9eb-1focal [3,924 B] 2022-03-08T23:04:39.891 INFO:teuthology.orchestra.run.smithi117.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11025-g7a44e9eb-1focal [50.5 MB] 2022-03-08T23:04:42.030 INFO:teuthology.orchestra.run.smithi117.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11025-g7a44e9eb-1focal [219 kB] 2022-03-08T23:04:42.032 INFO:teuthology.orchestra.run.smithi117.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11025-g7a44e9eb-1focal [69.5 kB] 2022-03-08T23:04:42.032 INFO:teuthology.orchestra.run.smithi117.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11025-g7a44e9eb-1focal [21.7 kB] 2022-03-08T23:04:42.120 INFO:teuthology.orchestra.run.smithi117.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11025-g7a44e9eb-1focal [12.7 MB] 2022-03-08T23:04:42.545 INFO:teuthology.orchestra.run.smithi117.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [78.8 kB] 2022-03-08T23:04:43.152 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 181 MB in 8s (22.3 MB/s) 2022-03-08T23:04:43.318 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-08T23:04:43.365 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-08T23:04:43.368 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-08T23:04:43.546 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T23:04:43.984 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../01-librbd1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:44.164 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librbd1 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T23:04:44.961 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../02-librados2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:45.096 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librados2 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T23:04:45.913 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs2. 2022-03-08T23:04:45.926 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:45.964 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:46.314 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rados. 2022-03-08T23:04:46.324 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:46.364 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:46.776 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-08T23:04:46.790 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:04:46.844 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:47.128 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cephfs. 2022-03-08T23:04:47.143 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:47.205 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:47.617 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-08T23:04:47.629 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:04:47.667 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:48.150 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-prettytable. 2022-03-08T23:04:48.163 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-08T23:04:48.210 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-08T23:04:48.552 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rbd. 2022-03-08T23:04:48.566 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:48.683 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:49.243 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-08T23:04:49.259 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-08T23:04:49.313 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T23:04:49.689 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-08T23:04:49.702 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-08T23:04:49.742 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T23:04:50.224 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-08T23:04:50.238 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-08T23:04:50.285 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T23:04:50.665 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librgw2. 2022-03-08T23:04:50.678 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../13-librgw2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:50.716 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:51.515 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rgw. 2022-03-08T23:04:51.524 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:51.565 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:51.944 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liboath0. 2022-03-08T23:04:51.959 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-08T23:04:52.012 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-08T23:04:52.574 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libradosstriper1. 2022-03-08T23:04:52.587 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:52.654 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:53.123 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-common. 2022-03-08T23:04:53.136 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:53.175 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:55.831 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-base. 2022-03-08T23:04:55.843 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:04:55.935 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:57.219 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cryptography. 2022-03-08T23:04:57.227 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-08T23:04:57.287 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T23:04:57.746 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-openssl. 2022-03-08T23:04:57.759 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-08T23:04:57.797 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-08T23:04:58.120 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-08T23:04:58.134 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:04:58.212 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:04:58.666 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-08T23:04:58.679 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-08T23:04:58.707 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-08T23:04:59.040 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-08T23:04:59.076 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-08T23:04:59.444 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jwt. 2022-03-08T23:04:59.458 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-08T23:04:59.498 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T23:04:59.920 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-logutils. 2022-03-08T23:04:59.933 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-08T23:04:59.972 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-08T23:05:00.416 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-08T23:05:00.432 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-08T23:05:00.487 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-08T23:05:00.871 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-mako. 2022-03-08T23:05:00.885 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T23:05:01.311 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-08T23:05:01.319 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-08T23:05:01.360 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-08T23:05:01.660 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-08T23:05:01.674 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-08T23:05:01.717 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T23:05:02.155 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webob. 2022-03-08T23:05:02.168 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-08T23:05:02.230 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-08T23:05:02.725 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-08T23:05:02.734 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-08T23:05:02.775 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T23:05:03.137 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bs4. 2022-03-08T23:05:03.150 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-08T23:05:03.195 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-08T23:05:03.531 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-waitress. 2022-03-08T23:05:03.546 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-08T23:05:03.599 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-08T23:05:05.664 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempita. 2022-03-08T23:05:05.676 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-08T23:05:05.732 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-08T23:05:06.044 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-paste. 2022-03-08T23:05:06.053 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-08T23:05:06.099 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T23:05:06.433 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-08T23:05:06.447 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-08T23:05:06.510 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T23:05:06.869 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-08T23:05:06.880 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-08T23:05:06.919 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-08T23:05:07.318 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webtest. 2022-03-08T23:05:07.327 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-08T23:05:07.368 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-08T23:05:07.893 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pecan. 2022-03-08T23:05:07.907 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-08T23:05:07.951 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T23:05:08.318 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-08T23:05:08.330 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-08T23:05:08.368 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T23:05:08.732 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-08T23:05:08.745 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:08.787 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:09.304 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr. 2022-03-08T23:05:09.315 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:09.362 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:09.911 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mon. 2022-03-08T23:05:09.924 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:09.965 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:11.314 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-osd. 2022-03-08T23:05:11.325 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:11.410 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:14.458 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph. 2022-03-08T23:05:14.470 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../45-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:14.507 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:14.993 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-fuse. 2022-03-08T23:05:15.006 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:15.087 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:15.718 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mds. 2022-03-08T23:05:15.732 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:15.792 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:16.386 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-08T23:05:16.399 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-08T23:05:16.444 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-08T23:05:16.730 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjs-underscore. 2022-03-08T23:05:16.742 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-08T23:05:16.779 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T23:05:17.075 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-08T23:05:17.086 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-08T23:05:17.124 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T23:05:17.687 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-routes. 2022-03-08T23:05:17.696 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-08T23:05:17.754 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-08T23:05:18.097 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-08T23:05:18.111 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:05:18.147 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:18.881 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-decorator. 2022-03-08T23:05:18.894 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-08T23:05:18.931 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T23:05:19.241 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-08T23:05:19.257 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-08T23:05:19.308 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T23:05:19.758 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-scipy. 2022-03-08T23:05:19.773 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-08T23:05:19.849 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-08T23:05:21.492 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-08T23:05:21.502 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-08T23:05:21.542 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T23:05:22.240 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-joblib. 2022-03-08T23:05:22.251 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-08T23:05:22.307 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-08T23:05:22.768 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn. 2022-03-08T23:05:22.782 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-08T23:05:22.826 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T23:05:23.693 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-08T23:05:23.704 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:05:23.793 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:25.495 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cachetools. 2022-03-08T23:05:25.509 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-08T23:05:25.570 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-08T23:05:25.935 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-08T23:05:25.948 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-08T23:05:25.993 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T23:05:26.380 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-08T23:05:26.393 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-08T23:05:26.440 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T23:05:26.759 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rsa. 2022-03-08T23:05:26.772 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-08T23:05:26.816 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-08T23:05:27.173 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-google-auth. 2022-03-08T23:05:27.186 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-08T23:05:27.232 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-08T23:05:27.795 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-blinker. 2022-03-08T23:05:27.808 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-08T23:05:27.861 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T23:05:28.156 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-08T23:05:28.170 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-08T23:05:28.215 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T23:05:28.647 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-08T23:05:28.657 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-08T23:05:28.705 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T23:05:29.151 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-websocket. 2022-03-08T23:05:29.163 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-08T23:05:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T23:05:29.652 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-08T23:05:29.664 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-08T23:05:29.710 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T23:05:30.307 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-08T23:05:30.318 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:05:30.361 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:30.764 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-08T23:05:30.776 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-08T23:05:30.814 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-08T23:05:31.251 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-08T23:05:31.260 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T23:05:31.309 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T23:05:31.624 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package jq. 2022-03-08T23:05:31.640 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T23:05:31.724 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T23:05:32.111 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-08T23:05:32.123 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-08T23:05:32.169 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T23:05:32.489 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package xmlstarlet. 2022-03-08T23:05:32.502 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-08T23:05:32.555 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-08T23:05:33.048 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-test. 2022-03-08T23:05:33.061 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:33.098 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:38.955 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-volume. 2022-03-08T23:05:38.969 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T23:05:39.020 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:39.422 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cephadm. 2022-03-08T23:05:39.439 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../78-cephadm_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:39.513 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:39.864 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-08T23:05:39.877 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:39.950 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:40.353 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package nvme-cli. 2022-03-08T23:05:40.362 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-08T23:05:40.417 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T23:05:41.002 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-08T23:05:41.017 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-08T23:05:41.059 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T23:05:41.404 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-attr. 2022-03-08T23:05:41.419 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-08T23:05:41.462 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-08T23:05:41.910 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webencodings. 2022-03-08T23:05:41.926 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-08T23:05:42.005 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T23:05:42.450 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-html5lib. 2022-03-08T23:05:42.462 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-08T23:05:42.527 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-08T23:05:43.019 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-08T23:05:43.033 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-08T23:05:43.076 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T23:05:43.606 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-packaging. 2022-03-08T23:05:43.623 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-08T23:05:43.664 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-08T23:05:44.140 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastescript. 2022-03-08T23:05:44.153 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-08T23:05:44.206 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-08T23:05:44.637 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pluggy. 2022-03-08T23:05:44.652 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-08T23:05:44.702 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-08T23:05:45.079 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-psutil. 2022-03-08T23:05:45.096 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-08T23:05:45.188 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T23:05:45.608 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-py. 2022-03-08T23:05:45.618 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-08T23:05:45.658 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T23:05:46.036 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-08T23:05:46.051 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-08T23:05:46.094 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T23:05:46.523 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-08T23:05:46.535 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-08T23:05:46.581 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T23:05:46.872 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pytest. 2022-03-08T23:05:46.887 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-08T23:05:46.930 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-08T23:05:47.360 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplejson. 2022-03-08T23:05:47.373 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-08T23:05:47.475 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T23:05:48.079 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package radosgw. 2022-03-08T23:05:48.198 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../95-radosgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:48.200 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:49.623 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package rbd-fuse. 2022-03-08T23:05:49.636 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T23:05:49.700 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:50.041 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-08T23:05:50.437 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T23:05:50.562 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T23:05:50.797 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:51.591 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user cephadm....done 2022-03-08T23:05:51.770 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-08T23:05:52.130 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-08T23:05:52.497 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-08T23:05:52.648 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T23:05:53.007 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-08T23:05:53.440 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T23:05:53.818 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-08T23:05:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T23:05:54.434 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T23:05:54.837 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:05:55.181 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T23:05:55.321 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-08T23:05:55.657 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T23:05:56.015 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T23:05:56.443 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T23:05:56.725 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T23:05:57.219 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T23:05:57.563 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-08T23:05:57.971 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-08T23:05:58.373 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T23:05:58.531 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T23:05:58.816 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T23:05:59.260 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T23:05:59.645 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T23:06:00.019 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:00.330 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-08T23:06:00.680 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T23:06:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-08T23:06:01.107 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T23:06:01.268 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T23:06:01.492 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T23:06:01.827 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T23:06:02.118 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-08T23:06:02.453 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T23:06:02.646 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-08T23:06:02.995 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-08T23:06:03.288 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T23:06:03.580 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T23:06:03.669 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-08T23:06:03.923 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-08T23:06:04.091 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T23:06:04.390 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-08T23:06:04.793 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T23:06:04.894 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T23:06:05.027 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-08T23:06:05.362 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T23:06:05.688 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librados2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:05.807 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-08T23:06:07.459 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:08.470 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:08.704 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T23:06:08.847 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-08T23:06:09.231 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-08T23:06:09.574 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-08T23:06:09.925 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-08T23:06:10.286 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:10.420 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T23:06:10.762 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:10.955 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T23:06:11.298 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T23:06:12.486 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librbd1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:12.637 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T23:06:12.830 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-08T23:06:13.072 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T23:06:13.515 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T23:06:13.641 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-08T23:06:13.934 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-08T23:06:14.226 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:14.386 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:14.618 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T23:06:14.618 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T23:06:15.596 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:15.731 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T23:06:16.055 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-08T23:06:16.480 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T23:06:16.874 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:17.058 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:17.224 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:17.342 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-08T23:06:17.654 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-08T23:06:17.952 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:18.139 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:18.321 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T23:06:19.498 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:20.147 INFO:teuthology.orchestra.run.smithi117.stdout:Adding group ceph....done 2022-03-08T23:06:20.549 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user ceph....done 2022-03-08T23:06:20.735 INFO:teuthology.orchestra.run.smithi117.stdout:Setting system user ceph properties....done 2022-03-08T23:06:20.779 INFO:teuthology.orchestra.run.smithi117.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-08T23:06:20.925 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-08T23:06:21.507 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-08T23:06:22.540 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:22.702 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:22.995 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T23:06:22.995 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T23:06:24.349 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:24.773 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-08T23:06:25.833 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:26.088 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T23:06:26.089 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T23:06:27.023 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:27.255 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T23:06:27.255 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T23:06:28.191 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:28.928 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T23:06:28.929 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T23:06:29.866 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:30.017 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:30.186 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:30.408 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T23:06:30.408 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T23:06:31.334 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:31.460 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:31.594 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T23:06:31.732 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-08T23:06:32.241 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T23:06:35.128 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T23:06:35.577 INFO:teuthology.orchestra.run.smithi117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-08T23:06:35.580 DEBUG:teuthology.parallel:result is None 2022-03-08T23:06:35.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-03-08T23:06:35.631 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-03-08T23:06:35.637 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:06:35.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T23:06:35.794 DEBUG:teuthology.orchestra.run.smithi117:> dpkg-query -W -f '${Version}' ceph 2022-03-08T23:06:35.813 INFO:teuthology.orchestra.run.smithi117.stdout:17.0.0-11025-g7a44e9eb-1focal 2022-03-08T23:06:35.814 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T23:06:35.814 INFO:teuthology.task.install:The correct ceph version 17.0.0-11025-g7a44e9eb-1focal is installed. 2022-03-08T23:06:35.814 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-08T23:06:35.815 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:06:35.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-08T23:06:35.871 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-08T23:06:35.871 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:06:35.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2022-03-08T23:06:35.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-08T23:06:35.896 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-08T23:06:35.897 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-03-08T23:06:35.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-08T23:06:35.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-08T23:06:35.967 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-08T23:06:35.981 INFO:tasks.radosbench:Beginning radosbench... 2022-03-08T23:06:35.982 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-08T23:06:35.982 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd pool create unique_pool_0 16 2022-03-08T23:06:37.688 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'unique_pool_0' created 2022-03-08T23:06:37.705 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-08T23:06:38.860 INFO:teuthology.orchestra.run.smithi117.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-08T23:06:39.891 DEBUG:teuthology.orchestra.run.smithi117:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-08T23:06:39.894 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:06:39.904 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch host label add '`hostname`' 'foo 2022-03-08T23:06:39.905 DEBUG:teuthology.orchestra.run.smithi117:> ceph' orch host ls '|' grep 'foo 2022-03-08T23:06:39.905 DEBUG:teuthology.orchestra.run.smithi117:> ceph' orch host label rm '`hostname`' 'foo 2022-03-08T23:06:39.905 DEBUG:teuthology.orchestra.run.smithi117:> ceph' orch host ls '|' grep -v 'foo 2022-03-08T23:06:39.906 DEBUG:teuthology.orchestra.run.smithi117:> ' 2022-03-08T23:06:40.280 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:hints = 1 2022-03-08T23:06:40.280 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-08T23:06:40.280 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Object prefix: benchmark_data_smithi117_49994 2022-03-08T23:06:40.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:06:40.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 0 0 0 0 0 0 - 0 2022-03-08T23:06:40.585 INFO:teuthology.orchestra.run.smithi117.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-08T23:06:40.609 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T23:06:40.620 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T23:06:40.621 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi117.front.sepia.ceph.com 2022-03-08T23:06:40.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T23:06:40.622 DEBUG:teuthology.orchestra.run.smithi117:> toolbox() { 2022-03-08T23:06:40.622 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T23:06:40.622 DEBUG:teuthology.orchestra.run.smithi117:> } 2022-03-08T23:06:40.623 DEBUG:teuthology.orchestra.run.smithi117:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:06:40.623 DEBUG:teuthology.orchestra.run.smithi117:> toolbox ceph orch osd rm 0 --force 2022-03-08T23:06:40.623 DEBUG:teuthology.orchestra.run.smithi117:> removed_pv="" 2022-03-08T23:06:40.624 DEBUG:teuthology.orchestra.run.smithi117:> while [ "$removed_pv" = "" ] 2022-03-08T23:06:40.624 DEBUG:teuthology.orchestra.run.smithi117:> do 2022-03-08T23:06:40.624 DEBUG:teuthology.orchestra.run.smithi117:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T23:06:40.624 DEBUG:teuthology.orchestra.run.smithi117:> sleep 3s 2022-03-08T23:06:40.625 DEBUG:teuthology.orchestra.run.smithi117:> done 2022-03-08T23:06:40.625 DEBUG:teuthology.orchestra.run.smithi117:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T23:06:40.625 DEBUG:teuthology.orchestra.run.smithi117:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T23:06:40.626 DEBUG:teuthology.orchestra.run.smithi117:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T23:06:40.626 DEBUG:teuthology.orchestra.run.smithi117:> zap_completion="0" 2022-03-08T23:06:40.626 DEBUG:teuthology.orchestra.run.smithi117:> while [ "$zap_completion" = "0" ] 2022-03-08T23:06:40.626 DEBUG:teuthology.orchestra.run.smithi117:> do 2022-03-08T23:06:40.627 DEBUG:teuthology.orchestra.run.smithi117:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T23:06:40.627 DEBUG:teuthology.orchestra.run.smithi117:> sleep 3s 2022-03-08T23:06:40.627 DEBUG:teuthology.orchestra.run.smithi117:> done 2022-03-08T23:06:40.627 DEBUG:teuthology.orchestra.run.smithi117:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T23:06:40.628 DEBUG:teuthology.orchestra.run.smithi117:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:06:40.628 DEBUG:teuthology.orchestra.run.smithi117:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:06:40.628 DEBUG:teuthology.orchestra.run.smithi117:> num_osd="0" 2022-03-08T23:06:40.628 DEBUG:teuthology.orchestra.run.smithi117:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T23:06:40.629 DEBUG:teuthology.orchestra.run.smithi117:> do 2022-03-08T23:06:40.629 DEBUG:teuthology.orchestra.run.smithi117:> echo "waiting for osd to come back up" 2022-03-08T23:06:40.629 DEBUG:teuthology.orchestra.run.smithi117:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:06:40.630 DEBUG:teuthology.orchestra.run.smithi117:> sleep 30s 2022-03-08T23:06:40.630 DEBUG:teuthology.orchestra.run.smithi117:> done 2022-03-08T23:06:40.630 DEBUG:teuthology.orchestra.run.smithi117:> ' 2022-03-08T23:06:40.641 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:06:40.642 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:06:40.642 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:06:41.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 1 16 1224 1208 75.4957 75.5 0.0126268 0.0131747 2022-03-08T23:06:42.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 2 16 2786 2770 86.5559 97.625 0.00918625 0.0114157 2022-03-08T23:06:43.120 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:06:43.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 3 16 4270 4254 88.618 92.75 0.0151151 0.011248 2022-03-08T23:06:43.893 INFO:teuthology.orchestra.run.smithi117.stderr:+ orig_num_osd=4 2022-03-08T23:06:43.893 INFO:teuthology.orchestra.run.smithi117.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T23:06:43.893 INFO:teuthology.orchestra.run.smithi117.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T23:06:44.007 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:06:44.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 4 16 5473 5457 85.2586 75.1875 0.0252937 0.0117063 2022-03-08T23:06:44.834 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:06:44.833413 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:06:44.835 INFO:tasks.rook.operator.smithi117.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:06:44.835 INFO:tasks.rook.operator.smithi117.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:06:44.835 INFO:tasks.rook.operator.smithi117.stdout:   ... // 3 identical fields 2022-03-08T23:06:44.836 INFO:tasks.rook.operator.smithi117.stdout:   Config: nil, 2022-03-08T23:06:44.836 INFO:tasks.rook.operator.smithi117.stdout:   Selection: {}, 2022-03-08T23:06:44.836 INFO:tasks.rook.operator.smithi117.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:06:44.836 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:06:44.836 INFO:tasks.rook.operator.smithi117.stdout:-  { 2022-03-08T23:06:44.837 INFO:tasks.rook.operator.smithi117.stdout:-  Name: "smithi117-lv-2", 2022-03-08T23:06:44.837 INFO:tasks.rook.operator.smithi117.stdout:-  Count: 1, 2022-03-08T23:06:44.837 INFO:tasks.rook.operator.smithi117.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:06:44.837 INFO:tasks.rook.operator.smithi117.stdout:-  }, 2022-03-08T23:06:44.838 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:06:44.838 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:06:44.838 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:06:44.838 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:06:44.839 INFO:tasks.rook.operator.smithi117.stdout:   Annotations: nil, 2022-03-08T23:06:44.839 INFO:tasks.rook.operator.smithi117.stdout:   Labels: nil, 2022-03-08T23:06:44.839 INFO:tasks.rook.operator.smithi117.stdout:   ... // 20 identical fields 2022-03-08T23:06:44.839 INFO:tasks.rook.operator.smithi117.stdout:  } 2022-03-08T23:06:44.839 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:06:44.833489 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:06:44.916 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:06:44.916518 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:06:44.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:06:44.991188 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:06:45.281 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 5 16 6557 6541 81.756 67.75 0.0179217 0.0121579 2022-03-08T23:06:46.282 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 6 16 6557 6541 68.1299 0 - 0.0121579 2022-03-08T23:06:47.282 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 7 16 6557 6541 58.397 0 - 0.0121579 2022-03-08T23:06:48.282 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 8 16 6557 6541 51.0973 0 - 0.0121579 2022-03-08T23:06:49.282 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 9 16 6557 6541 45.4197 0 - 0.0121579 2022-03-08T23:06:50.283 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 10 16 6557 6541 40.8776 0 - 0.0121579 2022-03-08T23:06:50.700 INFO:teuthology.orchestra.run.smithi117.stdout:Success 2022-03-08T23:06:50.715 INFO:teuthology.orchestra.run.smithi117.stderr:+ removed_pv= 2022-03-08T23:06:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' '' = '' ']' 2022-03-08T23:06:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get pv 2022-03-08T23:06:50.717 INFO:teuthology.orchestra.run.smithi117.stderr:++ grep Released 2022-03-08T23:06:50.717 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f1 -d ' ' 2022-03-08T23:06:50.992 INFO:teuthology.orchestra.run.smithi117.stderr:+ removed_pv= 2022-03-08T23:06:50.992 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 3s 2022-03-08T23:06:51.283 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 11 16 6557 6541 37.1612 0 - 0.0121579 2022-03-08T23:06:52.283 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 12 16 6557 6541 34.0644 0 - 0.0121579 2022-03-08T23:06:53.283 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 13 16 6557 6541 31.4439 0 - 0.0121579 2022-03-08T23:06:53.993 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' '' = '' ']' 2022-03-08T23:06:53.994 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get pv 2022-03-08T23:06:53.994 INFO:teuthology.orchestra.run.smithi117.stderr:++ grep Released 2022-03-08T23:06:53.995 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f1 -d ' ' 2022-03-08T23:06:54.046 INFO:teuthology.orchestra.run.smithi117.stderr:+ removed_pv= 2022-03-08T23:06:54.046 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 3s 2022-03-08T23:06:54.284 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 14 16 6557 6541 29.1979 0 - 0.0121579 2022-03-08T23:06:55.284 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 15 16 6957 6941 28.9179 2.5 0.0229075 0.0345563 2022-03-08T23:06:56.284 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 16 16 7645 7629 29.7978 43 0.0116823 0.0335334 2022-03-08T23:06:57.047 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' '' = '' ']' 2022-03-08T23:06:57.048 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f1 -d ' ' 2022-03-08T23:06:57.050 INFO:teuthology.orchestra.run.smithi117.stderr:++ grep Released 2022-03-08T23:06:57.053 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get pv 2022-03-08T23:06:57.284 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 17 16 8396 8380 30.8057 46.9375 0.057926 0.0323278 2022-03-08T23:06:57.312 INFO:teuthology.orchestra.run.smithi117.stderr:+ removed_pv= 2022-03-08T23:06:57.312 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 3s 2022-03-08T23:06:58.285 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 18 16 9212 9196 31.9274 51 0.0130133 0.0312985 2022-03-08T23:06:58.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:06:58.495063 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:06:59.285 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 19 16 10146 10130 33.3191 58.375 0.0234141 0.0299924 2022-03-08T23:07:00.285 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:07:00.284206+0000 min lat: 0.00155948 max lat: 9.63709 avg lat: 0.0287086 2022-03-08T23:07:00.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:07:00.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 20 16 11141 11125 34.7622 62.1875 0.0153233 0.0287086 2022-03-08T23:07:00.314 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' '' = '' ']' 2022-03-08T23:07:00.314 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get pv 2022-03-08T23:07:00.315 INFO:teuthology.orchestra.run.smithi117.stderr:++ grep Released 2022-03-08T23:07:00.315 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f1 -d ' ' 2022-03-08T23:07:00.521 INFO:teuthology.orchestra.run.smithi117.stderr:+ removed_pv=smithi117-lv-2 2022-03-08T23:07:00.522 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 3s 2022-03-08T23:07:01.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 21 16 11518 11502 34.2288 23.5625 0.0112805 0.0290987 2022-03-08T23:07:02.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 22 16 12256 12240 34.7693 46.125 0.0110735 0.0287419 2022-03-08T23:07:03.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 23 16 13220 13204 35.8769 60.25 0.0221718 0.0278543 2022-03-08T23:07:03.523 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' smithi117-lv-2 = '' ']' 2022-03-08T23:07:03.523 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get pv smithi117-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:07:03.750 INFO:teuthology.orchestra.run.smithi117.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-08T23:07:03.750 INFO:teuthology.orchestra.run.smithi117.stderr:++ echo smithi117-lv-2 2022-03-08T23:07:03.751 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f1 -d - 2022-03-08T23:07:03.752 INFO:teuthology.orchestra.run.smithi117.stderr:+ host=smithi117 2022-03-08T23:07:03.752 INFO:teuthology.orchestra.run.smithi117.stderr:+ toolbox ceph orch device zap smithi117 /dev/vg_nvme/lv_2 --force 2022-03-08T23:07:03.752 INFO:teuthology.orchestra.run.smithi117.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi117 /dev/vg_nvme/lv_2 --force 2022-03-08T23:07:04.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 24 15 13983 13968 36.3714 47.75 0.0202209 0.0274761 2022-03-08T23:07:04.488 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:07:05.287 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 25 16 14731 14715 36.7839 46.6875 0.0146683 0.0271733 2022-03-08T23:07:06.047 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/vg_nvme/lv_2 on smithi117 zapped 2022-03-08T23:07:06.080 INFO:teuthology.orchestra.run.smithi117.stderr:+ zap_completion=0 2022-03-08T23:07:06.080 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:07:06.081 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:07:06.266 INFO:teuthology.orchestra.run.smithi117.stderr:+ zap_completion= 2022-03-08T23:07:06.266 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 3s 2022-03-08T23:07:06.286 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 26 16 15512 15496 37.2463 48.8125 0.0179088 0.026836 2022-03-08T23:07:07.287 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 27 16 16399 16383 37.9199 55.4375 0.0138207 0.0263203 2022-03-08T23:07:08.287 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 28 16 17242 17226 38.4471 52.6875 0.0214041 0.0259993 2022-03-08T23:07:09.267 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' '' = 0 ']' 2022-03-08T23:07:09.268 INFO:teuthology.orchestra.run.smithi117.stderr:+ kubectl patch pv smithi117-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:07:09.287 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 29 16 17922 17906 38.5868 42.5 0.0496275 0.0258819 2022-03-08T23:07:09.869 INFO:teuthology.orchestra.run.smithi117.stdout:persistentvolume/smithi117-lv-2 patched 2022-03-08T23:07:09.871 INFO:teuthology.orchestra.run.smithi117.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:07:09.871 INFO:teuthology.orchestra.run.smithi117.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:07:10.170 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:07:10.289 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 30 16 18572 18556 38.6527 40.625 0.00888808 0.0258562 2022-03-08T23:07:11.289 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 31 15 19277 19262 38.8291 44.125 0.153379 0.0257037 2022-03-08T23:07:11.755 INFO:teuthology.orchestra.run.smithi117.stdout:['Success'] 2022-03-08T23:07:11.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:11.753217 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:07:11.767 INFO:tasks.rook.operator.smithi117.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:07:11.768 INFO:tasks.rook.operator.smithi117.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:07:11.768 INFO:tasks.rook.operator.smithi117.stdout:   ... // 3 identical fields 2022-03-08T23:07:11.768 INFO:tasks.rook.operator.smithi117.stdout:   Config: nil, 2022-03-08T23:07:11.768 INFO:tasks.rook.operator.smithi117.stdout:   Selection: {}, 2022-03-08T23:07:11.769 INFO:tasks.rook.operator.smithi117.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:07:11.769 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:07:11.769 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:07:11.769 INFO:tasks.rook.operator.smithi117.stdout:   {Name: "smithi117-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi117-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:07:11.769 INFO:tasks.rook.operator.smithi117.stdout:+  { 2022-03-08T23:07:11.770 INFO:tasks.rook.operator.smithi117.stdout:+  Name: "smithi117-lv-2", 2022-03-08T23:07:11.770 INFO:tasks.rook.operator.smithi117.stdout:+  Count: 1, 2022-03-08T23:07:11.770 INFO:tasks.rook.operator.smithi117.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:07:11.770 INFO:tasks.rook.operator.smithi117.stdout:+  }, 2022-03-08T23:07:11.771 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:07:11.771 INFO:tasks.rook.operator.smithi117.stdout:   }, 2022-03-08T23:07:11.771 INFO:tasks.rook.operator.smithi117.stdout:   Annotations: nil, 2022-03-08T23:07:11.771 INFO:tasks.rook.operator.smithi117.stdout:   Labels: nil, 2022-03-08T23:07:11.772 INFO:tasks.rook.operator.smithi117.stdout:   ... // 20 identical fields 2022-03-08T23:07:11.772 INFO:tasks.rook.operator.smithi117.stdout:  } 2022-03-08T23:07:11.787 INFO:teuthology.orchestra.run.smithi117.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:07:12.289 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 32 16 19975 19959 38.9769 43.5625 0.0137575 0.0256429 2022-03-08T23:07:12.324 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:12.319765 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:07:12.324 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:12.319780 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:07:12.324 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:12.319786 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:07:12.636 INFO:teuthology.orchestra.run.smithi117.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:07:12.639 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=0 2022-03-08T23:07:12.639 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 0 '!=' 4 ']' 2022-03-08T23:07:12.639 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:07:12.640 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:07:12.640 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:07:12.641 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:07:12.641 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:07:12.641 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:12.636911 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:07:13.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:13.080518 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:07:13.233 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:13.227177 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:07:13.289 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 33 16 20727 20711 39.2199 47 0.0148413 0.0254842 2022-03-08T23:07:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:07:14.091 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.086714 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:07:14.292 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 34 16 21260 21244 39.0441 33.3125 0.0247534 0.0255966 2022-03-08T23:07:14.318 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.315142 I | op-mon: start running mons 2022-03-08T23:07:14.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.397401 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:07:14.470 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.465782 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:07:14.480 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.477316 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:07:14.485 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:14.477508 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:07:15.292 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 35 16 21853 21837 38.9874 37.0625 0.0136426 0.025636 2022-03-08T23:07:15.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:15.750712 I | op-mon: targeting the mon count 1 2022-03-08T23:07:15.777 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:15.773900 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T23:07:15.778 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:15.773990 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:07:16.192 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:16.186802 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:07:16.292 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 36 16 22562 22546 39.1351 44.3125 0.0220466 0.0255382 2022-03-08T23:07:17.292 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 37 16 23315 23299 39.3493 47.0625 0.0242433 0.0253674 2022-03-08T23:07:17.445 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:17.440016 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:07:18.047 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:18.042107 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:07:18.053 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:18.049455 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:07:18.292 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 38 16 23973 23957 39.3959 41.125 0.0313991 0.0253709 2022-03-08T23:07:19.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 39 16 24776 24760 39.6725 50.1875 0.0141813 0.0251947 2022-03-08T23:07:20.076 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:20.069238 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:07:20.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:07:20.289198+0000 min lat: 0.00155948 max lat: 9.63709 avg lat: 0.0251108 2022-03-08T23:07:20.752 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:07:20.752 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 40 16 25491 25475 39.7978 44.6875 0.0203662 0.0251108 2022-03-08T23:07:21.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 41 16 26121 26105 39.7873 39.375 0.0143718 0.0251204 2022-03-08T23:07:22.079 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:07:22.073798 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:07:22.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 42 16 26845 26829 39.9173 45.25 0.0268495 0.0250396 2022-03-08T23:07:23.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 43 16 27072 27056 39.319 14.1875 0.0389157 0.0250251 2022-03-08T23:07:24.293 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 44 16 27522 27506 39.0645 28.125 0.00772577 0.0253953 2022-03-08T23:07:25.294 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 45 16 27604 27588 38.3104 5.125 0.0126165 0.026086 2022-03-08T23:07:26.295 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 46 16 28235 28219 38.3334 39.4375 0.0209399 0.0260776 2022-03-08T23:07:27.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 47 15 28900 28885 38.4034 41.625 0.0267565 0.0260239 2022-03-08T23:07:28.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 48 16 29662 29646 38.5941 47.5625 0.0171944 0.0259004 2022-03-08T23:07:29.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 49 16 30322 30306 38.6482 41.25 0.0203293 0.0258648 2022-03-08T23:07:30.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 50 16 31006 30990 38.7302 42.75 0.0527161 0.0258061 2022-03-08T23:07:31.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 51 16 31708 31692 38.831 43.875 0.0127065 0.0257425 2022-03-08T23:07:32.296 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 52 16 32249 32233 38.7344 33.8125 0.014816 0.0257877 2022-03-08T23:07:33.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 53 16 32493 32477 38.2913 15.25 0.0152394 0.0258319 2022-03-08T23:07:34.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 54 16 32563 32547 37.6633 4.375 0.0144699 0.0264383 2022-03-08T23:07:35.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 55 16 32672 32656 37.1024 6.8125 0.00463296 0.026375 2022-03-08T23:07:36.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 56 16 32672 32656 36.4399 0 - 0.026375 2022-03-08T23:07:37.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 57 16 32672 32656 35.8007 0 - 0.026375 2022-03-08T23:07:38.297 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 58 16 32672 32656 35.1835 0 - 0.026375 2022-03-08T23:07:39.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 59 16 32672 32656 34.5872 0 - 0.026375 2022-03-08T23:07:40.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:07:40.292644+0000 min lat: 0.00155948 max lat: 9.63709 avg lat: 0.026375 2022-03-08T23:07:40.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:07:40.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 60 16 32672 32656 34.0108 0 - 0.026375 2022-03-08T23:07:41.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 61 16 32739 32723 33.5219 0.697917 0.0248199 0.0298184 2022-03-08T23:07:42.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 62 16 33396 33380 33.6434 41.0625 0.016217 0.0297095 2022-03-08T23:07:43.298 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 63 16 34004 33988 33.7125 38 0.0292749 0.0296512 2022-03-08T23:07:44.299 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 64 16 34565 34549 33.7332 35.0625 0.0149409 0.0296355 2022-03-08T23:07:44.890 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:07:44.890 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:07:44.891 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:07:44.891 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:07:44.892 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:07:44.892 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:07:44.985 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:07:45.299 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 65 16 35152 35136 33.7786 36.6875 0.0278839 0.0295949 2022-03-08T23:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:07:46.299 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 66 16 35651 35635 33.7393 31.1875 0.0137678 0.0296305 2022-03-08T23:07:47.299 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 67 16 36209 36193 33.7562 34.875 0.0335653 0.0296111 2022-03-08T23:07:48.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 68 16 36776 36760 33.7809 35.4375 0.0268943 0.029593 2022-03-08T23:07:49.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 69 16 37106 37090 33.5901 20.625 0.0299 0.0297615 2022-03-08T23:07:50.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 70 16 37632 37616 33.5798 32.875 0.0704084 0.0297522 2022-03-08T23:07:51.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 71 16 38152 38136 33.5645 32.5 0.0271179 0.0297833 2022-03-08T23:07:52.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 72 16 38693 38677 33.5679 33.8125 0.0121266 0.0297821 2022-03-08T23:07:53.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 73 16 39336 39320 33.6585 40.1875 0.00822933 0.0296952 2022-03-08T23:07:54.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 74 16 40019 40003 33.7805 42.6875 0.0247073 0.0295833 2022-03-08T23:07:55.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 75 16 40280 40264 33.5476 16.3125 0.0270069 0.0297977 2022-03-08T23:07:56.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 76 16 40895 40879 33.6119 38.4375 0.0167762 0.0297431 2022-03-08T23:07:57.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 77 16 41539 41523 33.698 40.25 0.0145426 0.0296679 2022-03-08T23:07:58.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 78 16 42003 41987 33.6378 29 0.0497129 0.0297005 2022-03-08T23:07:59.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 79 16 42532 42516 33.6305 33.0625 0.0965042 0.0297005 2022-03-08T23:08:00.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:08:00.295518+0000 min lat: 0.00155948 max lat: 9.63709 avg lat: 0.0296791 2022-03-08T23:08:00.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:08:00.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 80 16 43098 43082 33.6522 35.375 0.0235752 0.0296791 2022-03-08T23:08:01.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 81 16 43608 43592 33.6302 31.875 0.0223205 0.0297249 2022-03-08T23:08:02.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 82 16 44051 44035 33.5577 27.6875 0.0171945 0.0297913 2022-03-08T23:08:03.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 83 16 44653 44637 33.6067 37.625 0.0660321 0.0297453 2022-03-08T23:08:04.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 84 16 45238 45222 33.6418 36.5625 0.0264375 0.0297135 2022-03-08T23:08:05.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 85 16 45807 45791 33.6644 35.5625 0.0191777 0.0296967 2022-03-08T23:08:06.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 86 16 46314 46298 33.6414 31.6875 0.0325035 0.0297137 2022-03-08T23:08:07.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 87 16 46838 46822 33.6311 32.75 0.0216326 0.0296898 2022-03-08T23:08:08.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 88 16 47272 47256 33.5571 27.125 0.0143863 0.0297917 2022-03-08T23:08:09.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 89 16 47854 47838 33.5888 36.375 0.0249252 0.0297626 2022-03-08T23:08:10.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 90 16 48504 48488 33.6669 40.625 0.0274471 0.0296939 2022-03-08T23:08:11.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 91 16 49065 49049 33.6822 35.0625 0.00729633 0.0296295 2022-03-08T23:08:12.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 92 16 49078 49062 33.3249 0.8125 0.00315002 0.0296231 2022-03-08T23:08:13.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 93 16 49078 49062 32.9666 0 - 0.0296231 2022-03-08T23:08:14.129 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.124435 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:08:14.129 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.124456 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:08:14.130 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.124464 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:08:14.190 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.185964 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:08:14.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 94 16 49078 49062 32.6159 0 - 0.0296231 2022-03-08T23:08:15.000 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.992108 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:15.000 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:14.992436 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:15.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 95 16 49078 49062 32.2726 0 - 0.0296231 2022-03-08T23:08:15.708 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:15.703877 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:08:15.920 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:08:15.921 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:08:15.921 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:08:15.922 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:08:15.922 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:08:15.922 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:08:16.129 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:08:16.139 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:16.134970 I | op-mon: start running mons 2022-03-08T23:08:16.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 96 16 49078 49062 31.9365 0 - 0.0296231 2022-03-08T23:08:16.775 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:16.771682 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:08:17.052 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.045558 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:08:17.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.097878 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:17.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.098367 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:08:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:08:17.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 97 16 49078 49062 31.6073 0 - 0.0296231 2022-03-08T23:08:17.375 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.369917 I | op-mon: targeting the mon count 1 2022-03-08T23:08:17.385 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.374461 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:08:17.889 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.885406 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:08:17.890 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:17.885455 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:08:18.216 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.211747 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:08:18.216 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.211776 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:08:18.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 98 16 49078 49062 31.2847 0 - 0.0296231 2022-03-08T23:08:18.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.559737 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:08:18.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.559759 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:08:18.932 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.927833 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:08:18.932 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:18.927854 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:08:19.295 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:19.291294 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:08:19.295 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:19.291322 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:08:19.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 99 16 49078 49062 30.9687 0 - 0.0296231 2022-03-08T23:08:19.638 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:19.634767 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:08:19.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:19.634786 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:08:20.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:08:20.297504+0000 min lat: 0.00155948 max lat: 9.63709 avg lat: 0.0296231 2022-03-08T23:08:20.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:08:20.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 100 16 49078 49062 30.6591 0 - 0.0296231 2022-03-08T23:08:20.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.675928 I | op-mon: mon "a" endpoint is [v2:10.106.34.234:3300,v1:10.106.34.234:6789] 2022-03-08T23:08:20.707 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.704181 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:08:20.714 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.711273 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:20.717 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.711479 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:20.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.730405 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:08:20.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.738598 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:08:20.751 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.741261 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:08:20.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:20.747066 I | op-mon: mons running: [a] 2022-03-08T23:08:21.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:21.250078 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:08:21.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:21.250105 I | op-mon: mons created: 1 2022-03-08T23:08:21.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 101 16 49168 49152 30.4112 0.625 0.00349232 0.0327195 2022-03-08T23:08:21.669 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:21.665779 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:08:21.680 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:21.673777 I | op-mon: mons running: [a] 2022-03-08T23:08:22.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:22.157332 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:08:22.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:22.157368 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:08:22.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 102 16 49168 49152 30.1131 0 - 0.0327195 2022-03-08T23:08:22.721 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:22.717499 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:08:23.179 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:23.175293 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:08:23.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 103 16 49168 49152 29.8207 0 - 0.0327195 2022-03-08T23:08:23.607 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:23.603294 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:08:24.099 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.095948 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:08:24.100 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.095972 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:08:24.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 104 16 49168 49152 29.534 0 - 0.0327195 2022-03-08T23:08:24.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.540769 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:08:24.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.964581 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:08:24.969 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.964622 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:08:24.969 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:24.964633 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:08:25.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 105 16 49168 49152 29.2527 0 - 0.0327195 2022-03-08T23:08:25.475 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:25.471646 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:08:25.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:25.511504 I | op-mgr: start running mgr 2022-03-08T23:08:25.515 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:25.511553 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:08:26.035 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.031827 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:08:26.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.041568 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:08:26.047 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.043871 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:08:26.241 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.234333 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:08:26.242 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.234366 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:08:26.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 106 16 49168 49152 28.9768 0 - 0.0327195 2022-03-08T23:08:26.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.336449 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:08:26.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339414 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-1" 2022-03-08T23:08:26.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339448 I | op-osd: OSD PVC "smithi117-lv-1-data-07skck" already exists 2022-03-08T23:08:26.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339460 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-3" 2022-03-08T23:08:26.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339470 I | op-osd: OSD PVC "smithi117-lv-3-data-06h6fm" already exists 2022-03-08T23:08:26.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339480 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-4" 2022-03-08T23:08:26.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339490 I | op-osd: OSD PVC "smithi117-lv-4-data-0lbbtm" already exists 2022-03-08T23:08:26.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.339502 I | op-osd: creating 1 new PVCs for device set "smithi117-lv-2" 2022-03-08T23:08:26.350 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.347629 I | op-osd: successfully provisioned PVC "smithi117-lv-2-data-05fjqf" 2022-03-08T23:08:26.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.379276 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:08:26.393 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.379302 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:08:26.393 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.379307 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:08:26.393 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.379312 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-05fjqf" 2022-03-08T23:08:26.708 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.705043 I | op-osd: started OSD provisioning job for PVC "smithi117-lv-2-data-05fjqf" 2022-03-08T23:08:26.709 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.705062 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:08:26.709 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.705066 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:08:26.884 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:26.880827 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-05fjqf is "starting" 2022-03-08T23:08:27.300 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 107 16 49168 49152 28.706 0 - 0.0327195 2022-03-08T23:08:27.613 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:27.610387 I | op-mgr: successful modules: prometheus 2022-03-08T23:08:27.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:27.614890 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:08:27.621 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:27.616239 I | op-mgr: successful modules: dashboard 2022-03-08T23:08:27.979 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:27.975827 I | op-mgr: successful modules: balancer 2022-03-08T23:08:28.205 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:28.201765 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:08:28.208 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:28.205386 I | op-osd: updating OSD 1 on PVC "smithi117-lv-3-data-06h6fm" 2022-03-08T23:08:28.209 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:28.205407 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:08:28.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 108 16 49168 49152 28.4402 0 - 0.0327195 2022-03-08T23:08:29.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:29.261491 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:08:29.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 109 16 49313 49297 28.2624 1.13281 0.00370514 0.0339526 2022-03-08T23:08:29.583 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:29.580437 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:08:29.617 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:29.614846 I | op-osd: updating OSD 2 on PVC "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:08:29.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:29.614876 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:08:30.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 110 16 49347 49331 28.0248 2.125 0.0323808 0.0356711 2022-03-08T23:08:31.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 111 16 49895 49879 28.0809 34.25 0.025574 0.0356018 2022-03-08T23:08:31.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:31.478908 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:08:31.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:31.485305 I | op-osd: updating OSD 3 on PVC "smithi117-lv-1-data-07skck" 2022-03-08T23:08:31.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:08:31.485330 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:08:32.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 112 16 50507 50491 28.1716 38.25 0.0167075 0.0354857 2022-03-08T23:08:33.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 113 16 51123 51107 28.263 38.5 0.0324751 0.0353731 2022-03-08T23:08:34.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 114 16 51747 51731 28.3571 39 0.0174332 0.0352395 2022-03-08T23:08:35.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 115 16 52363 52347 28.4453 38.5 0.0349887 0.0351455 2022-03-08T23:08:36.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 116 16 52979 52963 28.5319 38.5 0.017175 0.0350395 2022-03-08T23:08:37.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 117 16 53536 53520 28.5856 34.8125 0.0205163 0.0349302 2022-03-08T23:08:38.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 118 16 53999 53983 28.5885 28.9375 0.0145236 0.0349703 2022-03-08T23:08:39.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 119 16 54602 54586 28.665 37.6875 0.0703497 0.0348742 2022-03-08T23:08:40.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:08:40.299492+0000 min lat: 0.00155948 max lat: 9.74898 avg lat: 0.0347861 2022-03-08T23:08:40.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:08:40.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 120 16 55201 55185 28.7381 37.4375 0.0357885 0.0347861 2022-03-08T23:08:41.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 121 16 55775 55759 28.797 35.875 0.0123759 0.0347152 2022-03-08T23:08:42.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 122 16 56149 56133 28.7525 23.375 0.0159873 0.0346789 2022-03-08T23:08:43.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 123 16 56597 56581 28.7464 28 0.0343483 0.0347782 2022-03-08T23:08:44.301 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 124 16 57060 57044 28.7479 28.9375 0.0517931 0.0347725 2022-03-08T23:08:45.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 125 16 57214 57198 28.5949 9.625 0.0267756 0.0347629 2022-03-08T23:08:46.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 126 16 57412 57396 28.4662 12.375 0.0325194 0.0351215 2022-03-08T23:08:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:08:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:08:47.209 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:08:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:08:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:08:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:08:47.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 127 16 57872 57856 28.4684 28.75 0.0656196 0.0351126 2022-03-08T23:08:47.640 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:08:48.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 128 16 58255 58239 28.433 23.9375 0.00306316 0.0350975 2022-03-08T23:08:48.839 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:08:48.839 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:08:49.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 129 16 58278 58262 28.2237 1.4375 0.00450644 0.0350877 2022-03-08T23:08:50.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 130 16 58278 58262 28.0066 0 - 0.0350877 2022-03-08T23:08:51.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 131 16 58278 58262 27.7928 0 - 0.0350877 2022-03-08T23:08:52.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 132 16 58278 58262 27.5823 0 - 0.0350877 2022-03-08T23:08:53.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 133 16 58278 58262 27.3749 0 - 0.0350877 2022-03-08T23:08:54.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 134 16 58278 58262 27.1706 0 - 0.0350877 2022-03-08T23:08:55.302 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 135 16 58278 58262 26.9694 0 - 0.0350877 2022-03-08T23:08:56.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 136 16 58278 58262 26.7711 0 - 0.0350877 2022-03-08T23:08:57.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 137 16 58278 58262 26.5757 0 - 0.0350877 2022-03-08T23:08:58.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 138 16 58278 58262 26.3831 0 - 0.0350877 2022-03-08T23:08:59.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 139 16 58278 58262 26.1933 0 - 0.0350877 2022-03-08T23:09:00.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:00.140221 I | op-osd: OSD orchestration status for node smithi117-lv-2-data-05fjqf is "orchestrating" 2022-03-08T23:09:00.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:09:00.301651+0000 min lat: 0.00155948 max lat: 9.74898 avg lat: 0.0350877 2022-03-08T23:09:00.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:09:00.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 140 16 58278 58262 26.0062 0 - 0.0350877 2022-03-08T23:09:00.538 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:00.535875 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-05fjqf is "orchestrating" 2022-03-08T23:09:01.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 141 16 58565 58549 25.949 1.49479 0.0317107 0.0385275 2022-03-08T23:09:02.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 142 16 59106 59090 26.0043 33.8125 0.0278517 0.0384464 2022-03-08T23:09:03.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 143 16 59642 59626 26.0567 33.5 0.0237919 0.0383679 2022-03-08T23:09:04.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 144 16 60146 60130 26.0945 31.5 0.0384002 0.0383134 2022-03-08T23:09:05.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 145 16 60600 60584 26.1102 28.375 0.0171764 0.0382897 2022-03-08T23:09:06.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 146 16 61171 61155 26.1758 35.6875 0.0206982 0.0381949 2022-03-08T23:09:07.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 147 16 61709 61693 26.2264 33.625 0.0317 0.0381218 2022-03-08T23:09:08.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 148 16 62269 62253 26.2857 35 0.03833 0.0380352 2022-03-08T23:09:09.289 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:09.227883 I | op-osd: OSD orchestration status for PVC smithi117-lv-2-data-05fjqf is "completed" 2022-03-08T23:09:09.290 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:09.227910 I | op-osd: creating OSD 0 on PVC "smithi117-lv-2-data-05fjqf" 2022-03-08T23:09:09.290 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:09.227919 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-05fjqf" 2022-03-08T23:09:09.303 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 149 16 62385 62369 26.1579 7.25 0.019552 0.0380284 2022-03-08T23:09:10.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 150 16 62731 62715 26.1277 21.625 0.019383 0.0382546 2022-03-08T23:09:11.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 151 16 63190 63174 26.1446 28.6875 0.0295158 0.0382412 2022-03-08T23:09:12.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 152 16 63632 63616 26.1543 27.625 0.0443698 0.038218 2022-03-08T23:09:12.536 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:12.503211 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:09:12.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:12.503230 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:12.807 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:12.805026 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:13.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 153 16 64081 64065 26.1668 28.0625 0.0211844 0.038193 2022-03-08T23:09:14.012 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:14.009712 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:09:14.092 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:14.090123 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:09:14.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 154 16 64506 64490 26.1693 26.5625 0.0484319 0.0382005 2022-03-08T23:09:15.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 155 16 64926 64910 26.1698 26.25 0.0248848 0.038203 2022-03-08T23:09:16.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 156 16 65397 65381 26.1908 29.4375 0.0395195 0.0381737 2022-03-08T23:09:17.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 157 16 65758 65742 26.1677 22.5625 0.0406231 0.038181 2022-03-08T23:09:18.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 158 16 66302 66286 26.2172 34 0.030473 0.0381337 2022-03-08T23:09:18.839 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:09:18.839 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:09:18.840 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:09:18.841 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:09:18.841 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:09:18.842 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:09:19.152 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:19.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 159 16 66811 66795 26.2524 31.8125 0.0195344 0.0380824 2022-03-08T23:09:20.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:09:20.303674+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0380517 2022-03-08T23:09:20.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:09:20.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 160 16 67284 67268 26.273 29.5625 0.0430327 0.0380517 2022-03-08T23:09:21.022 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:09:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:09:21.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 161 16 67751 67735 26.2911 29.1875 0.0270981 0.0380267 2022-03-08T23:09:22.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 162 16 68143 68127 26.2801 24.5 0.0305161 0.0380437 2022-03-08T23:09:23.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 163 16 68621 68605 26.3021 29.875 0.0179869 0.0380122 2022-03-08T23:09:24.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 164 16 69064 69048 26.3105 27.6875 0.0417942 0.0379969 2022-03-08T23:09:25.304 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 165 16 69522 69506 26.3245 28.625 0.0291378 0.0379786 2022-03-08T23:09:26.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 166 16 69974 69958 26.3361 28.25 0.0588346 0.0379577 2022-03-08T23:09:27.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 167 16 70425 70409 26.3472 28.1875 0.0393632 0.0379396 2022-03-08T23:09:28.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 168 16 70933 70917 26.3793 31.75 0.03492 0.0379009 2022-03-08T23:09:29.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 169 16 71226 71210 26.3316 18.3125 0.0169698 0.0379524 2022-03-08T23:09:30.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 170 16 71350 71334 26.2223 7.75 0.00455282 0.0379184 2022-03-08T23:09:31.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 171 16 71350 71334 26.0689 0 - 0.0379184 2022-03-08T23:09:32.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 172 16 71350 71334 25.9174 0 - 0.0379184 2022-03-08T23:09:33.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 173 16 71350 71334 25.7676 0 - 0.0379184 2022-03-08T23:09:34.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 174 16 71350 71334 25.6195 0 - 0.0379184 2022-03-08T23:09:35.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 175 16 71350 71334 25.4731 0 - 0.0379184 2022-03-08T23:09:36.240 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.238773 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:09:36.240 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.238798 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:09:36.240 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.238812 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:09:36.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 176 16 71350 71334 25.3284 0 - 0.0379184 2022-03-08T23:09:36.657 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.656682 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:09:36.884 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.883365 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:36.887 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:36.883742 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:37.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 177 16 71350 71334 25.1853 0 - 0.0379184 2022-03-08T23:09:37.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.338290 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:09:37.499 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.498524 I | op-mon: start running mons 2022-03-08T23:09:37.534 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.533660 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:09:37.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.620905 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:09:37.628 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.627355 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:37.628 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:37.627501 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:38.305 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 178 16 71350 71334 25.0438 0 - 0.0379184 2022-03-08T23:09:38.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:38.557157 I | op-mon: targeting the mon count 1 2022-03-08T23:09:38.598 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:38.597231 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:38.990 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:38.989203 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:38.990 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:38.989224 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:39.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 179 16 71350 71334 24.9039 0 - 0.0379184 2022-03-08T23:09:39.323 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:39.322616 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:39.323 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:39.322643 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:39.666 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:39.665439 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:39.667 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:39.665479 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:40.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:40.170042 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:40.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:40.170062 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:09:40.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:09:40.305557+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0379184 2022-03-08T23:09:40.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:09:40.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 180 16 71350 71334 24.7655 0 - 0.0379184 2022-03-08T23:09:40.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:40.489220 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:09:40.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:40.489240 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:09:41.009 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:41.008311 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:09:41.009 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:41.008335 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:09:41.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 181 16 71601 71585 24.7154 1.42614 0.0163473 0.0404523 2022-03-08T23:09:42.128 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.127428 I | op-mon: mon "a" endpoint is [v2:10.106.34.234:3300,v1:10.106.34.234:6789] 2022-03-08T23:09:42.191 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.189104 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.34.234:6789"]}] data:a=10.106.34.234:6789 mapping:{"node":{"a":{"Name":"smithi117","Hostname":"smithi117","Address":"172.21.15.117"}}} maxMonId:0] 2022-03-08T23:09:42.202 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.200855 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:42.202 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.201022 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:42.215 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.214427 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:09:42.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.245909 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:09:42.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.245939 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:09:42.260 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:42.259844 I | op-mon: mons running: [a] 2022-03-08T23:09:42.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 182 16 72090 72074 24.7475 30.5625 0.0518517 0.0403988 2022-03-08T23:09:43.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 183 16 72565 72549 24.7745 29.6875 0.0364339 0.0403549 2022-03-08T23:09:43.348 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:43.280678 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:09:43.348 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:43.280702 I | op-mon: mons created: 1 2022-03-08T23:09:44.076 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:44.072816 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:09:44.300 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:44.299766 I | op-mon: mons running: [a] 2022-03-08T23:09:44.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 184 16 73022 73006 24.795 28.5625 0.0337274 0.0403211 2022-03-08T23:09:45.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:45.107254 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:09:45.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:45.107286 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:09:45.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 185 16 73467 73451 24.8113 27.8125 0.0603965 0.0402946 2022-03-08T23:09:45.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:45.836201 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:09:46.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 186 16 73894 73878 24.8214 26.6875 0.0330531 0.0402804 2022-03-08T23:09:46.557 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:46.555811 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:09:47.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 187 16 74136 74120 24.7695 15.125 0.0199711 0.0403635 2022-03-08T23:09:48.007 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:47.914973 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:09:48.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 188 16 74567 74551 24.7811 26.9375 0.0421913 0.0403457 2022-03-08T23:09:49.178 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:49.177156 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:09:49.179 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:49.177179 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:09:49.306 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 189 16 75017 75001 24.7987 28.125 0.0607309 0.0403138 2022-03-08T23:09:49.936 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:49.934176 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:09:50.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 190 16 75464 75448 24.8152 27.9375 0.0330315 0.0402898 2022-03-08T23:09:50.592 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:50.590691 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:09:50.592 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:50.590717 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:09:50.592 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:50.590723 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:09:51.023 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:09:51.024 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:09:51.024 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:09:51.024 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:09:51.025 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:09:51.025 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:09:51.142 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:51.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 191 16 75876 75860 24.8201 25.75 0.0310291 0.0402802 2022-03-08T23:09:51.389 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:51.386537 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:09:51.418 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:51.417529 I | op-mgr: start running mgr 2022-03-08T23:09:51.419 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:51.417556 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:09:52.030 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=3 2022-03-08T23:09:52.030 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:09:52.241 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.240872 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:09:52.249 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.246164 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:09:52.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.250998 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:09:52.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 192 16 76269 76253 24.8188 24.5625 0.041143 0.0402829 2022-03-08T23:09:52.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.384805 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:09:52.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.384829 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:09:52.410 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.406863 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:09:52.415 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410375 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-1" 2022-03-08T23:09:52.415 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410430 I | op-osd: OSD PVC "smithi117-lv-1-data-07skck" already exists 2022-03-08T23:09:52.415 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410442 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-3" 2022-03-08T23:09:52.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410450 I | op-osd: OSD PVC "smithi117-lv-3-data-06h6fm" already exists 2022-03-08T23:09:52.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410458 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-4" 2022-03-08T23:09:52.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410465 I | op-osd: OSD PVC "smithi117-lv-4-data-0lbbtm" already exists 2022-03-08T23:09:52.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410491 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi117-lv-2" 2022-03-08T23:09:52.417 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.410504 I | op-osd: OSD PVC "smithi117-lv-2-data-05fjqf" already exists 2022-03-08T23:09:52.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425747 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:09:52.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425772 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:09:52.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425778 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:09:52.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425783 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-05fjqf" 2022-03-08T23:09:52.428 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425787 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:09:52.428 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:52.425790 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:09:53.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 193 16 76493 76477 24.7627 14 0.0386592 0.0403 2022-03-08T23:09:53.845 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:53.844199 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:09:53.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:53.850004 I | op-osd: updating OSD 0 on PVC "smithi117-lv-2-data-05fjqf" 2022-03-08T23:09:53.851 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:53.850032 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-2-data-05fjqf" 2022-03-08T23:09:54.249 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:54.247255 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:09:54.252 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:54.249464 I | op-mgr: successful modules: prometheus 2022-03-08T23:09:54.255 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:54.252321 I | op-mgr: successful modules: dashboard 2022-03-08T23:09:54.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 194 16 76493 76477 24.6351 0 - 0.0403 2022-03-08T23:09:54.517 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:54.515908 I | op-mgr: successful modules: balancer 2022-03-08T23:09:55.124 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:55.123023 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:09:55.129 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:55.128800 I | op-osd: updating OSD 1 on PVC "smithi117-lv-3-data-06h6fm" 2022-03-08T23:09:55.129 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:55.128827 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-3-data-06h6fm" 2022-03-08T23:09:55.307 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 195 16 76493 76477 24.5087 0 - 0.0403 2022-03-08T23:09:55.693 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:55.692355 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:09:56.310 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 196 16 76493 76477 24.3834 0 - 0.0403 2022-03-08T23:09:56.975 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:56.973906 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:09:56.981 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:56.980357 I | op-osd: updating OSD 2 on PVC "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:09:56.981 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:56.980382 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-4-data-0lbbtm" 2022-03-08T23:09:57.310 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 197 16 76493 76477 24.2596 0 - 0.0403 2022-03-08T23:09:58.310 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 198 16 76583 76567 24.1655 1.125 0.0612369 0.0413701 2022-03-08T23:09:59.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:59.006608 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:09:59.016 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:59.015469 I | op-osd: updating OSD 3 on PVC "smithi117-lv-1-data-07skck" 2022-03-08T23:09:59.016 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:59.015492 I | op-osd: OSD will have its main bluestore block on "smithi117-lv-1-data-07skck" 2022-03-08T23:09:59.310 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 199 16 76879 76863 24.137 18.5 0.0476267 0.0414173 2022-03-08T23:09:59.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:59.959298 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:09:59.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:09:59.959320 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:10:00.310 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:10:00.310209+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.041465 2022-03-08T23:10:00.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:10:00.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 200 16 77182 77166 24.111 18.9375 0.0231134 0.041465 2022-03-08T23:10:01.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 201 16 77472 77456 24.0812 18.125 0.0971428 0.0415152 2022-03-08T23:10:02.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 202 16 77786 77770 24.0591 19.625 0.0890165 0.0415534 2022-03-08T23:10:03.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 203 16 78127 78111 24.0456 21.3125 0.0510609 0.0415789 2022-03-08T23:10:04.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 204 16 78460 78444 24.0297 20.8125 0.10759 0.0416036 2022-03-08T23:10:05.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 205 16 78760 78744 24.004 18.75 0.104306 0.0416483 2022-03-08T23:10:06.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 206 16 78920 78904 23.936 10 0.0138425 0.0416871 2022-03-08T23:10:07.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 207 16 79193 79177 23.9028 17.0625 0.0849291 0.0418266 2022-03-08T23:10:08.311 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 208 16 79500 79484 23.8801 19.1875 0.0598062 0.0418645 2022-03-08T23:10:09.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 209 16 79816 79800 23.8603 19.75 0.030228 0.0419013 2022-03-08T23:10:10.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 210 16 80145 80129 23.8446 20.5625 0.0512996 0.0419261 2022-03-08T23:10:11.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 211 16 80375 80359 23.7997 14.375 0.0547787 0.0420051 2022-03-08T23:10:12.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 212 16 80697 80681 23.7824 20.125 0.0640053 0.0420364 2022-03-08T23:10:13.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 213 16 80873 80857 23.7224 11 0.0406405 0.042144 2022-03-08T23:10:14.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 214 16 81204 81188 23.7082 20.6875 0.0415196 0.0421706 2022-03-08T23:10:15.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 215 16 81502 81486 23.6845 18.625 0.0789876 0.0422094 2022-03-08T23:10:16.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 216 16 81702 81686 23.6328 12.5 0.0936162 0.0423034 2022-03-08T23:10:17.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 217 16 82016 82000 23.6143 19.625 0.0313384 0.0423359 2022-03-08T23:10:18.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 218 16 82276 82260 23.5805 16.25 0.228889 0.0423904 2022-03-08T23:10:19.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 219 16 82489 82473 23.5336 13.3125 0.114548 0.0424783 2022-03-08T23:10:20.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:10:20.312191+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0424803 2022-03-08T23:10:20.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:10:20.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 220 16 82523 82507 23.4363 2.125 0.00733208 0.0424803 2022-03-08T23:10:21.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 221 16 82666 82650 23.3707 8.9375 0.0629226 0.0427799 2022-03-08T23:10:22.031 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:10:22.031 INFO:teuthology.orchestra.run.smithi117.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:10:22.032 INFO:teuthology.orchestra.run.smithi117.stdout:waiting for osd to come back up 2022-03-08T23:10:22.032 INFO:teuthology.orchestra.run.smithi117.stderr:++ toolbox ceph osd stat 2022-03-08T23:10:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:10:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:++ cut -f3 -d ' ' 2022-03-08T23:10:22.154 INFO:teuthology.orchestra.run.smithi117.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:10:22.312 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 222 16 82953 82937 23.3462 17.9375 0.0141219 0.0428189 2022-03-08T23:10:23.254 INFO:teuthology.orchestra.run.smithi117.stderr:+ num_osd=4 2022-03-08T23:10:23.255 INFO:teuthology.orchestra.run.smithi117.stderr:+ sleep 30s 2022-03-08T23:10:23.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 223 16 83223 83207 23.3172 16.875 0.0367026 0.0428766 2022-03-08T23:10:24.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 224 16 83489 83473 23.2873 16.625 0.131435 0.0429239 2022-03-08T23:10:25.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 225 16 83763 83747 23.2599 17.125 0.0534303 0.0429747 2022-03-08T23:10:26.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 226 16 84058 84042 23.2386 18.4375 0.0407346 0.0430215 2022-03-08T23:10:27.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 227 16 84332 84316 23.2116 17.125 0.0327224 0.0430711 2022-03-08T23:10:28.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 228 15 84663 84648 23.2008 20.75 0.0406708 0.0430927 2022-03-08T23:10:29.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 229 16 84867 84851 23.1549 12.6875 0.0527027 0.0431766 2022-03-08T23:10:30.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 230 16 85173 85157 23.1374 19.125 0.184698 0.0432056 2022-03-08T23:10:31.313 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 231 16 85366 85350 23.0894 12.0625 0.0228035 0.0432976 2022-03-08T23:10:32.314 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 232 15 85691 85676 23.0776 20.375 0.038855 0.0433229 2022-03-08T23:10:33.314 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 233 16 86001 85985 23.0615 19.3125 0.0257307 0.0433505 2022-03-08T23:10:34.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 234 16 86321 86305 23.0484 20 0.0730937 0.0433739 2022-03-08T23:10:35.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 235 16 86600 86584 23.0245 17.4375 0.0648431 0.0434214 2022-03-08T23:10:36.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 236 16 86778 86762 22.9741 11.125 0.0289436 0.0435154 2022-03-08T23:10:37.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 237 16 86990 86974 22.933 13.25 0.0794123 0.0435905 2022-03-08T23:10:38.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 238 16 87293 87277 22.9162 18.9375 0.019976 0.0436274 2022-03-08T23:10:39.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 239 16 87594 87578 22.899 18.8125 0.077483 0.0436583 2022-03-08T23:10:40.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:10:40.315194+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0436947 2022-03-08T23:10:40.316 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:10:40.316 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 240 16 87890 87874 22.8807 18.5 0.0518123 0.0436947 2022-03-08T23:10:41.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 241 16 88171 88155 22.8586 17.5625 0.0514061 0.0437376 2022-03-08T23:10:42.315 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 242 16 88459 88443 22.8386 18 0.12634 0.0437734 2022-03-08T23:10:43.316 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 243 16 88706 88690 22.8081 15.4375 0.00739804 0.0438033 2022-03-08T23:10:44.316 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 244 16 88926 88910 22.771 13.75 0.0444034 0.0439041 2022-03-08T23:10:45.316 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 245 16 89236 89220 22.7571 19.375 0.0352055 0.0439334 2022-03-08T23:10:46.317 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 246 16 89537 89521 22.741 18.8125 0.0686664 0.043963 2022-03-08T23:10:47.317 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 247 16 89849 89833 22.7278 19.5 0.0422074 0.0439882 2022-03-08T23:10:48.317 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 248 16 90165 90149 22.7158 19.75 0.0513471 0.0440114 2022-03-08T23:10:49.317 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 249 16 90387 90371 22.6803 13.875 0.0658387 0.0440796 2022-03-08T23:10:50.317 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 250 16 90669 90653 22.6601 17.625 0.0360902 0.04412 2022-03-08T23:10:51.318 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 251 16 90825 90809 22.6086 9.75 0.0323514 0.0442208 2022-03-08T23:10:52.318 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 252 16 91127 91111 22.5938 18.875 0.0482429 0.0442496 2022-03-08T23:10:53.256 INFO:teuthology.orchestra.run.smithi117.stderr:+ '[' 4 '!=' 4 ']' 2022-03-08T23:10:53.258 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:10:53.286 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch status 2022-03-08T23:10:53.318 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 253 16 91449 91433 22.584 20.125 0.0339606 0.0442694 2022-03-08T23:10:54.318 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 254 16 91748 91732 22.5687 18.6875 0.106515 0.0442969 2022-03-08T23:10:55.318 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 255 16 92051 92035 22.5544 18.9375 0.0663081 0.0443279 2022-03-08T23:10:56.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 256 16 92243 92227 22.5129 12 0.329545 0.0444063 2022-03-08T23:10:57.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 257 16 92523 92507 22.4934 17.5 0.0142999 0.0444426 2022-03-08T23:10:57.950 INFO:teuthology.orchestra.run.smithi117.stdout:Backend: rook 2022-03-08T23:10:57.951 INFO:teuthology.orchestra.run.smithi117.stdout:Available: Yes 2022-03-08T23:10:57.968 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ps 2022-03-08T23:10:58.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 258 16 92682 92666 22.4447 9.9375 0.0343561 0.044545 2022-03-08T23:10:59.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 259 16 92939 92923 22.4201 16.0625 0.0108626 0.0445903 2022-03-08T23:10:59.354 INFO:teuthology.orchestra.run.smithi117.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:10:59.354 INFO:teuthology.orchestra.run.smithi117.stdout:crashcollector.crash smithi117 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:10:59.355 INFO:teuthology.orchestra.run.smithi117.stdout:mgr.a smithi117 running (9m) 0s ago 9m - - 4b871cde5e1a 2022-03-08T23:10:59.355 INFO:teuthology.orchestra.run.smithi117.stdout:mon.a smithi117 running (9m) 0s ago 9m - - 4b871cde5e1a 2022-03-08T23:10:59.355 INFO:teuthology.orchestra.run.smithi117.stdout:osd.0 smithi117 running (108s) 0s ago 108s - - 4b871cde5e1a 2022-03-08T23:10:59.355 INFO:teuthology.orchestra.run.smithi117.stdout:osd.1 smithi117 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:10:59.356 INFO:teuthology.orchestra.run.smithi117.stdout:osd.2 smithi117 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:10:59.356 INFO:teuthology.orchestra.run.smithi117.stdout:osd.3 smithi117 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:10:59.386 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls 2022-03-08T23:11:00.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:11:00.321706+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0446427 2022-03-08T23:11:00.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:11:00.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 260 16 93176 93160 22.3908 14.8125 0.131239 0.0446427 2022-03-08T23:11:01.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 261 16 93398 93382 22.3582 13.875 0.0187142 0.0447129 2022-03-08T23:11:02.322 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 262 16 93641 93625 22.3308 15.1875 0.0741808 0.044772 2022-03-08T23:11:02.655 INFO:teuthology.orchestra.run.smithi117.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:11:02.655 INFO:teuthology.orchestra.run.smithi117.stdout:crash 1/1 1s ago 7m * 2022-03-08T23:11:02.655 INFO:teuthology.orchestra.run.smithi117.stdout:mgr 1/1 1s ago 9m count:1 2022-03-08T23:11:02.656 INFO:teuthology.orchestra.run.smithi117.stdout:mon 1/1 1s ago 9m count:1 2022-03-08T23:11:02.656 INFO:teuthology.orchestra.run.smithi117.stdout:osd 8 1s ago 7m 2022-03-08T23:11:02.656 INFO:teuthology.orchestra.run.smithi117.stdout:osd.all-available-devices 0 1s ago - * 2022-03-08T23:11:02.707 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch host ls 2022-03-08T23:11:03.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 263 16 93865 93849 22.2991 14 0.0282038 0.0448296 2022-03-08T23:11:04.137 INFO:teuthology.orchestra.run.smithi117.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:11:04.137 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 172.21.15.117/smithi117 2022-03-08T23:11:04.138 INFO:teuthology.orchestra.run.smithi117.stdout:1 hosts in cluster 2022-03-08T23:11:04.153 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch device ls 2022-03-08T23:11:04.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 264 15 94064 94049 22.262 12.5 0.0909948 0.0449011 2022-03-08T23:11:05.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 265 16 94304 94288 22.2344 14.9375 0.0463804 0.0449643 2022-03-08T23:11:05.646 INFO:teuthology.orchestra.run.smithi117.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:11:05.647 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:11:05.647 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:11:05.647 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:11:05.648 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:11:05.706 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch apply rgw foo 2022-03-08T23:11:06.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 266 16 94546 94530 22.2077 15.125 0.0739926 0.0450177 2022-03-08T23:11:06.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:06.803370 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:11:06.808 INFO:teuthology.orchestra.run.smithi117.stdout:Created 2022-03-08T23:11:06.857 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch apply mds foo 2022-03-08T23:11:06.938 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:06.933012 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:07.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 267 16 94777 94761 22.1785 14.4375 0.0386944 0.0450739 2022-03-08T23:11:07.671 INFO:teuthology.orchestra.run.smithi117.stdout:Created 2022-03-08T23:11:07.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:07.665542 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:11:07.714 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch apply rbd-mirror 2022-03-08T23:11:07.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:07.715443 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:08.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 268 16 94951 94935 22.1364 10.875 0.106465 0.0451632 2022-03-08T23:11:08.703 INFO:teuthology.orchestra.run.smithi117.stdout:Success 2022-03-08T23:11:08.719 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:08.715858 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:08.747 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch apply nfs foo --port 12777 2022-03-08T23:11:09.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 269 16 95120 95104 22.0933 10.5625 0.0421085 0.0452447 2022-03-08T23:11:09.468 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.466280 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:09.554 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.549791 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:11:09.554 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.549820 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:09.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.554678 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T23:11:09.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.554708 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T23:11:09.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:09.554719 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T23:11:10.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 270 16 95352 95336 22.0652 14.5 0.0654999 0.0453094 2022-03-08T23:11:11.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 271 16 95525 95509 22.0237 10.8125 0.108435 0.0453909 2022-03-08T23:11:11.334 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:11.332491 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:11.509 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:11.507174 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T23:11:12.316 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:12.314365 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:11:12.323 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 272 16 95681 95665 21.9786 9.75 0.0713077 0.0454788 2022-03-08T23:11:12.579 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:12.577538 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:11:12.669 INFO:teuthology.orchestra.run.smithi117.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:11:12.670 INFO:teuthology.orchestra.run.smithi117.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:11:12.748 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-08T23:11:12.765 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-08T23:11:12.766 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-08T23:11:13.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 273 16 95805 95789 21.9264 7.75 0.239513 0.0455822 2022-03-08T23:11:14.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 274 16 95994 95978 21.8895 11.8125 0.0617993 0.04567 2022-03-08T23:11:15.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 275 16 96248 96232 21.8676 15.875 0.0530735 0.0457204 2022-03-08T23:11:15.383 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:15.381507 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:11:16.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 276 16 96484 96468 21.8419 14.75 0.118746 0.045767 2022-03-08T23:11:17.324 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 277 16 96661 96645 21.8029 11.0625 0.0196362 0.0458532 2022-03-08T23:11:18.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 278 16 96929 96913 21.7848 16.75 0.0149694 0.0458901 2022-03-08T23:11:19.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 279 16 97183 97167 21.7636 15.875 0.0718921 0.045937 2022-03-08T23:11:20.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:11:20.323727+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0459902 2022-03-08T23:11:20.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:11:20.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 280 16 97411 97395 21.7367 14.25 0.0407634 0.0459902 2022-03-08T23:11:21.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 281 15 97621 97606 21.7063 13.1875 0.0458102 0.0460556 2022-03-08T23:11:22.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 282 16 97862 97846 21.6825 15 0.0259741 0.0461106 2022-03-08T23:11:23.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 283 16 98067 98051 21.6512 12.8125 0.0636951 0.0461697 2022-03-08T23:11:24.325 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 284 16 98268 98252 21.6192 12.5625 0.053387 0.046242 2022-03-08T23:11:25.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 285 16 98473 98457 21.5883 12.8125 0.097278 0.0463115 2022-03-08T23:11:26.033 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:26.030306 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-08T23:11:26.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:26.053977 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-08T23:11:26.063 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:26.060193 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-08T23:11:26.108 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:26.091349 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-08T23:11:26.108 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:26.097673 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-08T23:11:26.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 286 16 98704 98688 21.5633 14.4375 0.0315346 0.0463476 2022-03-08T23:11:27.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 287 16 98901 98885 21.531 12.3125 0.0555504 0.0464305 2022-03-08T23:11:28.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 288 16 99104 99088 21.5003 12.6875 0.00903085 0.0464865 2022-03-08T23:11:28.662 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:28.648710 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-08T23:11:29.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 289 16 99302 99286 21.4687 12.375 0.0372382 0.046567 2022-03-08T23:11:30.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 290 16 99560 99544 21.4503 16.125 0.0470594 0.0466049 2022-03-08T23:11:31.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 291 16 99814 99798 21.4311 15.875 0.0251322 0.0466505 2022-03-08T23:11:32.326 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 292 16 100113 100097 21.4217 18.6875 0.0502812 0.0466715 2022-03-08T23:11:32.401 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:32.398273 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-08T23:11:32.419 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:32.409766 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-08T23:11:32.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:32.425466 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:11:32.452 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:32.446630 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-08T23:11:32.458 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:32.453951 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-08T23:11:33.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 293 16 100351 100335 21.3994 14.875 0.125711 0.0467187 2022-03-08T23:11:34.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 294 16 100614 100598 21.3825 16.4375 0.0350765 0.0467546 2022-03-08T23:11:34.651 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:34.648276 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:11:34.752 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:34.749740 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:11:34.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:34.749767 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:11:34.775 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:34.772867 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:35.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 295 16 100864 100848 21.363 15.625 0.0917685 0.0467951 2022-03-08T23:11:36.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 296 16 101098 101082 21.3402 14.625 0.0589429 0.0468483 2022-03-08T23:11:37.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 297 16 101377 101361 21.3271 17.4375 0.0307321 0.0468785 2022-03-08T23:11:38.100 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:38.098251 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:38.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 298 16 101609 101593 21.3042 14.5 0.0285581 0.0469279 2022-03-08T23:11:39.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 299 16 101891 101875 21.2919 17.625 0.0388222 0.0469559 2022-03-08T23:11:39.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:39.337301 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:11:39.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:39.337325 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:39.600 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:39.597262 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-08T23:11:40.252 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:40.245455 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:40.327 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:11:40.325744+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0469963 2022-03-08T23:11:40.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:11:40.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 300 16 102122 102106 21.269 14.4375 0.0444225 0.0469963 2022-03-08T23:11:41.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 301 16 102269 102253 21.2289 9.1875 0.0617146 0.0470943 2022-03-08T23:11:41.520 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:41.518093 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:11:42.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 302 16 102487 102471 21.2037 13.625 0.0487882 0.0471512 2022-03-08T23:11:42.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:42.444709 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-08T23:11:42.485 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:42.444743 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:11:43.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 303 16 102688 102672 21.1752 12.5625 0.0318272 0.0472073 2022-03-08T23:11:44.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 304 16 102969 102953 21.1633 17.5625 0.128489 0.0472395 2022-03-08T23:11:44.806 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:44.803962 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:11:44.823 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:44.820188 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:45.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 305 16 103177 103161 21.1365 13 0.0298232 0.0472989 2022-03-08T23:11:46.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 306 16 103453 103437 21.1238 17.25 0.0640565 0.0473317 2022-03-08T23:11:46.834 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:46.828559 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:11:47.328 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 307 16 103697 103681 21.1046 15.25 0.06927 0.0473723 2022-03-08T23:11:48.020 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:48.017141 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:11:48.199 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:48.197046 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:48.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 308 16 103940 103924 21.0854 15.1875 0.0390621 0.0474137 2022-03-08T23:11:48.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:48.337611 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:11:48.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:48.337636 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:49.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:49.107131 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:49.260 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:49.254625 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:11:49.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 309 16 104176 104160 21.0649 14.75 0.0845025 0.0474582 2022-03-08T23:11:50.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 310 16 104339 104323 21.0298 10.1875 0.0469884 0.0475318 2022-03-08T23:11:50.471 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:50.468781 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:11:51.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 311 16 104546 104530 21.0038 12.9375 0.0446501 0.0475982 2022-03-08T23:11:51.479 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:51.475557 I | op-mds: setting mds config flags 2022-03-08T23:11:51.479 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:51.475587 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:52.311 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:52.304368 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:52.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:52.313543 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:11:52.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 312 16 104786 104770 20.9846 15 0.043586 0.0476398 2022-03-08T23:11:52.575 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:52.570369 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:11:52.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:52.667224 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:11:53.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 313 16 105002 104986 20.9606 13.5 0.0340902 0.0476917 2022-03-08T23:11:54.318 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:54.315476 I | op-mds: setting mds config flags 2022-03-08T23:11:54.319 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:54.315507 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:54.329 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 314 16 105227 105211 20.9387 14.0625 0.0422422 0.047747 2022-03-08T23:11:54.810 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:54.779078 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:55.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 315 16 105469 105453 20.9202 15.125 0.0509241 0.0477909 2022-03-08T23:11:55.470 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:55.461701 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:56.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 316 16 105763 105747 20.9122 18.375 0.052189 0.0478064 2022-03-08T23:11:56.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:56.882835 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:11:56.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:56.882859 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:57.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 317 16 105966 105950 20.8862 12.6875 0.0470287 0.0478633 2022-03-08T23:11:57.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:11:57.560832 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:58.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 318 16 106177 106161 20.862 13.1875 0.0965681 0.0479235 2022-03-08T23:11:59.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 319 16 106368 106352 20.834 11.9375 0.0920057 0.047986 2022-03-08T23:12:00.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:12:00.327870+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0480143 2022-03-08T23:12:00.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:12:00.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 320 16 106638 106622 20.8216 16.875 0.0440418 0.0480143 2022-03-08T23:12:01.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 321 16 106854 106838 20.7988 13.5 0.0378454 0.0480639 2022-03-08T23:12:01.355 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:01.351373 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:01.508 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:01.503868 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:02.330 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 322 16 107088 107072 20.7797 14.625 0.0476379 0.0481114 2022-03-08T23:12:03.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 323 16 107299 107283 20.7561 13.1875 0.0471166 0.0481681 2022-03-08T23:12:03.638 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:03.634979 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:04.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 324 16 107527 107511 20.7361 14.25 0.0349854 0.0482131 2022-03-08T23:12:05.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 325 16 107766 107750 20.7182 14.9375 0.0337846 0.0482559 2022-03-08T23:12:05.531 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:05.524932 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:05.532 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:05.524960 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:06.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 326 16 108013 107997 20.702 15.4375 0.0837709 0.0482918 2022-03-08T23:12:07.040 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:07.031675 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:07.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 327 16 108219 108203 20.6781 12.875 0.126681 0.0483463 2022-03-08T23:12:08.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 328 16 108409 108393 20.6512 11.875 0.0394982 0.0484139 2022-03-08T23:12:09.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 329 16 108674 108658 20.6388 16.5625 0.0583639 0.0484434 2022-03-08T23:12:10.331 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 330 16 108903 108887 20.6196 14.3125 0.139356 0.0484858 2022-03-08T23:12:11.332 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 331 16 109134 109118 20.6009 14.4375 0.027565 0.0485299 2022-03-08T23:12:11.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:11.493193 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:12.332 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 332 16 109376 109360 20.5844 15.125 0.0385249 0.0485703 2022-03-08T23:12:12.661 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:12.658269 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:13.332 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 333 16 109622 109606 20.5688 15.375 0.0526654 0.0486062 2022-03-08T23:12:14.332 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 334 16 109878 109862 20.5551 16 0.0991947 0.04864 2022-03-08T23:12:15.333 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 335 16 110131 110115 20.5409 15.8125 0.0597145 0.048673 2022-03-08T23:12:16.032 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:16.026954 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:16.232 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:16.226399 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:16.232 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:16.226424 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:16.333 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 336 16 110358 110342 20.522 14.1875 0.0351513 0.0487184 2022-03-08T23:12:16.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:16.837423 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:17.333 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 337 16 110529 110513 20.4928 10.6875 0.283862 0.0487574 2022-03-08T23:12:18.333 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 338 16 110685 110669 20.461 9.75 0.0389475 0.0488629 2022-03-08T23:12:19.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 339 16 110890 110874 20.4384 12.8125 0.211164 0.0488901 2022-03-08T23:12:20.150 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:20.145710 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:20.321 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:20.315566 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:20.333 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:12:20.330816+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0489569 2022-03-08T23:12:20.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:12:20.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 340 16 111129 111113 20.4223 14.9375 0.0700939 0.0489569 2022-03-08T23:12:21.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 341 16 111440 111424 20.4194 19.4375 0.111421 0.0489565 2022-03-08T23:12:22.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 342 16 111635 111619 20.3953 12.1875 0.0986228 0.0490189 2022-03-08T23:12:22.465 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:22.461597 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:22.557 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:22.550956 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:22.559 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:22.550979 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:23.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:23.134402 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:23.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 343 16 111920 111904 20.3878 17.8125 0.0590446 0.0490341 2022-03-08T23:12:24.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 344 16 112109 112093 20.3628 11.8125 0.0937392 0.0490966 2022-03-08T23:12:25.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 345 16 112347 112331 20.3469 14.875 0.02731 0.049138 2022-03-08T23:12:25.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:25.982868 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:26.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:26.313637 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:26.334 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 346 16 112624 112608 20.3381 17.3125 0.0255921 0.0491598 2022-03-08T23:12:27.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 347 16 112965 112949 20.341 21.3125 0.035361 0.049151 2022-03-08T23:12:28.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 348 16 113353 113337 20.3522 24.25 0.0422509 0.0491266 2022-03-08T23:12:28.608 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:28.604455 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:28.690 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:28.685689 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:28.690 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:28.685782 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:29.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 349 16 113742 113726 20.3635 24.3125 0.0207255 0.0490985 2022-03-08T23:12:29.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:29.398043 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:30.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 350 16 113989 113973 20.3494 15.4375 0.0245072 0.0491287 2022-03-08T23:12:31.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 351 16 114312 114296 20.349 20.1875 0.104406 0.0491298 2022-03-08T23:12:31.799 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:31.795542 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:32.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 352 16 114578 114562 20.3384 16.625 0.016158 0.0491589 2022-03-08T23:12:32.448 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:32.444161 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:33.335 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 353 16 114952 114936 20.347 23.375 0.0879377 0.0491372 2022-03-08T23:12:34.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 354 16 115306 115290 20.352 22.125 0.00925469 0.0491247 2022-03-08T23:12:34.833 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:34.829198 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:34.941 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:34.932386 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:34.941 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:34.932414 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:35.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 355 16 115668 115652 20.3584 22.625 0.0704318 0.049109 2022-03-08T23:12:35.596 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:35.590686 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:36.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 356 16 115917 115901 20.3449 15.5625 0.0838721 0.049138 2022-03-08T23:12:37.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 357 16 116194 116178 20.3364 17.3125 0.0486447 0.0491624 2022-03-08T23:12:38.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 358 15 116545 116530 20.3411 22 0.0955692 0.049149 2022-03-08T23:12:38.641 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:38.636751 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:39.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 359 16 116953 116937 20.3553 25.4375 0.0152808 0.0491174 2022-03-08T23:12:39.926 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:39.922060 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:40.336 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:2022-03-08T23:12:40.332768+0000 min lat: 0.00155948 max lat: 12.9538 avg lat: 0.0491185 2022-03-08T23:12:40.337 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:12:40.337 INFO:tasks.radosbench.radosbench.a.smithi117.stdout: 360 16 117026 117010 20.3114 4.5625 0.00747696 0.0491185 2022-03-08T23:12:41.337 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Total time run: 360.719 2022-03-08T23:12:41.338 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Total writes made: 117026 2022-03-08T23:12:41.338 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Write size: 65536 2022-03-08T23:12:41.338 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Object size: 65536 2022-03-08T23:12:41.339 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Bandwidth (MB/sec): 20.2765 2022-03-08T23:12:41.339 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Stddev Bandwidth: 15.6732 2022-03-08T23:12:41.340 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Max bandwidth (MB/sec): 97.625 2022-03-08T23:12:41.340 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Min bandwidth (MB/sec): 0 2022-03-08T23:12:41.341 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Average IOPS: 324 2022-03-08T23:12:41.341 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Stddev IOPS: 250.778 2022-03-08T23:12:41.342 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Max IOPS: 1562 2022-03-08T23:12:41.342 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Min IOPS: 0 2022-03-08T23:12:41.343 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Average Latency(s): 0.0493096 2022-03-08T23:12:41.343 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Stddev Latency(s): 0.296679 2022-03-08T23:12:41.344 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Max latency(s): 12.9538 2022-03-08T23:12:41.344 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Min latency(s): 0.00155948 2022-03-08T23:12:41.345 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Cleaning up (deleting benchmark objects) 2022-03-08T23:12:41.993 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:41.986986 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:42.066 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:42.055734 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:42.066 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:42.055761 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:42.697 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:42.692260 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:45.807 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:45.802209 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:48.609 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:48.604441 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:12:49.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:49.799921 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:12:51.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:51.101264 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:12:51.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:51.101318 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:12:51.934 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:51.929430 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:12:55.958 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:12:55.946469 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:13:01.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:01.076008 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:13:03.176 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:03.170680 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:13:03.335 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:03.329559 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:13:03.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:03.329730 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:13:04.005 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:03.997510 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:13:06.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:06.799836 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:13:17.061 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:17.055962 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:13:19.119 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:19.114148 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:13:19.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:19.248631 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:13:19.258 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:19.248654 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:13:19.879 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:19.871137 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:13:23.151 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:23.142038 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:13:43.638 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:43.631907 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:13:45.867 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:45.857127 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:13:45.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:45.964036 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:13:45.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:45.964058 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:13:46.541 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:46.534989 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:13:49.248 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:13:49.240479 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:14:30.220 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:30.212042 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:14:32.018 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:32.009326 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:14:32.120 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:32.112709 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:14:32.120 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:32.112733 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:14:32.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:32.877155 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:14:36.173 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:14:36.163946 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:15:58.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:15:58.099071 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:16:00.184 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:16:00.170356 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:16:00.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:16:00.269236 I | ceph-object-controller: ceph object store gateway service running at 10.96.69.58 2022-03-08T23:16:00.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:16:00.269260 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:16:01.016 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:16:01.003365 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:16:03.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:16:03.960159 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:18:16.886 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Removed 117026 objects 2022-03-08T23:18:16.888 INFO:tasks.radosbench.radosbench.a.smithi117.stdout:Clean up completed and total clean up time :335.531 2022-03-08T23:18:19.849 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-08T23:18:19.850 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-08T23:18:21.633 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'unique_pool_0' removed 2022-03-08T23:18:21.656 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-08T23:18:21.747 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-08T23:18:21.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-08T23:18:21.799 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:18:21.809 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:18:22.454 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:18:22.454 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:11:11.000000Z", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:22.336518Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:18:22.336518Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:22.336518Z", "running": 0, "size": 1}}] 2022-03-08T23:18:22.474 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch rm mds.foo 2022-03-08T23:18:22.959 INFO:teuthology.orchestra.run.smithi117.stdout:Removed cephfilesystems/foo 2022-03-08T23:18:22.980 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch rm rbd-mirror 2022-03-08T23:18:23.026 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:23.012683 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:18:23.498 INFO:teuthology.orchestra.run.smithi117.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:18:23.517 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:23.503528 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T23:18:23.528 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch rm rgw.foo 2022-03-08T23:18:24.461 INFO:teuthology.orchestra.run.smithi117.stdout:Removed cephobjectstores/foo 2022-03-08T23:18:24.513 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:24.498081 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:18:24.530 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:18:24.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:24.851109 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:18:26.841 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:26.791850 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:18:26.842 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:26.791875 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:18:26.842 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:26.791882 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:18:26.975 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:18:26.976 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:11:11.000000Z", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:26.369242Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:26.369242Z", "running": 0, "size": 1}}] 2022-03-08T23:18:30.270 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:30.255851 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:18:30.809 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:30.795153 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:18:31.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:31.636276 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:18:31.651 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:31.636304 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:18:31.651 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:31.636311 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:18:32.094 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:32.079167 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:18:32.070+0000 7f73ff186340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:18:32.094 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:32.070+0000 7f73ff186340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:18:32.094 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:32.070+0000 7f73ff186340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:18:32.095 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:18:32.095 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:32.079192 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:18:32.070+0000 7f73ff186340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:18:32.095 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:32.070+0000 7f73ff186340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:18:32.095 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:32.070+0000 7f73ff186340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:18:32.096 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:18:32.329 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:32.314749 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:18:32.330 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:32.314789 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:18:34.257 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:34.243042 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:18:37.035 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:18:38.943 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:18:38.943 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:11:11.000000Z", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:37.605130Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:37.605130Z", "running": 0, "size": 1}}] 2022-03-08T23:18:39.432 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:39.415467 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:18:39.432 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:39.415487 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:18:42.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:42.141481 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:18:42.573 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:42.558280 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:18:42.573 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:42.558309 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:18:44.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.603336 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:18:44.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.603379 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:18:44.644 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.629523 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:18:44.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.631947 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:18:44.647 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.632340 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:18:44.655 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:44.640648 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:18:47.825 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:47.811034 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:18:48.404 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:48.389232 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:18:48.377+0000 7f89c450e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:18:48.404 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:48.377+0000 7f89c450e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:18:48.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:48.377+0000 7f89c450e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:18:48.405 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:18:48.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:48.389256 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:18:48.377+0000 7f89c450e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:18:48.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:48.377+0000 7f89c450e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:18:48.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:18:48.377+0000 7f89c450e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:18:48.406 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:18:48.708 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:48.693183 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:18:48.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:18:48.856410 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi117": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:18:48.967 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:18:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:18:50.020 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:11:11.000000Z", "last_refresh": "2022-03-08T23:18:49.445219Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:18:49.445219Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:18:49.445219Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:18:49.445219Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:49.445219Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:49.445219Z", "running": 0, "size": 1}}] 2022-03-08T23:19:00.043 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:00.776 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:00.777 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:11:11.000000Z", "last_refresh": "2022-03-08T23:19:00.572622Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:00.572622Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:00.572622Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:00.572622Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:00.572622Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:00.572622Z", "running": 0, "size": 1}}] 2022-03-08T23:19:10.800 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:11.448 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:11.449 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:19:11.346212Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:11.346212Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:11.346212Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:11.346212Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:11.346212Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:11.346212Z", "running": 0, "size": 1}}] 2022-03-08T23:19:21.471 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:22.097 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:22.098 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:19:21.987970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:21.987970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:21.987970Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:21.987970Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:21.987970Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:21.987970Z", "running": 0, "size": 1}}] 2022-03-08T23:19:32.122 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:32.831 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:32.831 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:19:32.634484Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:32.634484Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:32.634484Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:32.634484Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:32.634484Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:32.634484Z", "running": 0, "size": 1}}] 2022-03-08T23:19:42.851 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:43.441 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:43.441 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:19:43.337241Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:43.337241Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:43.337241Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:43.337241Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:43.337241Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:43.337241Z", "running": 0, "size": 1}}] 2022-03-08T23:19:53.462 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:19:54.116 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:19:54.117 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:19:53.939604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:19:53.939604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:19:53.939604Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:19:53.939604Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:53.939604Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:53.939604Z", "running": 0, "size": 1}}] 2022-03-08T23:20:04.143 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:20:05.185 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:20:05.185 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:20:04.632765Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:20:04.632765Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:20:04.632765Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:20:04.632765Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:04.632765Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:04.632765Z", "running": 0, "size": 1}}] 2022-03-08T23:20:15.207 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:20:15.899 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:20:15.899 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:20:15.755314Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:20:15.755314Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:20:15.755314Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:20:15.755314Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:15.755314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:15.755314Z", "running": 0, "size": 1}}] 2022-03-08T23:20:25.923 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:20:26.615 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:20:26.615 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:20:26.450228Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:20:26.450228Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:20:26.450228Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:20:26.450228Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:26.450228Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:26.450228Z", "running": 0, "size": 1}}] 2022-03-08T23:20:36.639 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:20:37.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:20:37.561 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:20:37.453036Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:20:37.453036Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:20:37.453036Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:20:37.453036Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:37.453036Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:37.453036Z", "running": 0, "size": 1}}] 2022-03-08T23:20:47.584 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:20:48.261 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:20:48.262 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:20:48.130960Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:20:48.130960Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:20:48.130960Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:20:48.130960Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:48.130960Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:48.130960Z", "running": 0, "size": 1}}] 2022-03-08T23:20:58.285 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:01.644 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:01.645 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:01.502540Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:01.502540Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:01.502540Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:01.502540Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:01.502540Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:01.502540Z", "running": 0, "size": 1}}] 2022-03-08T23:21:11.666 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:12.319 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:12.319 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:12.182463Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:12.182463Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:12.182463Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:12.182463Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:12.182463Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:12.182463Z", "running": 0, "size": 1}}] 2022-03-08T23:21:22.340 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:22.952 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:22.952 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:22.847634Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:22.847634Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:22.847634Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:22.847634Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:22.847634Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:22.847634Z", "running": 0, "size": 1}}] 2022-03-08T23:21:32.975 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:33.579 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:33.580 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:33.467544Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:33.467544Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:33.467544Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:33.467544Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:33.467544Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:33.467544Z", "running": 0, "size": 1}}] 2022-03-08T23:21:43.600 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:44.222 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:44.222 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:44.113022Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:44.113022Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:44.113022Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:44.113022Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:44.113022Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:44.113022Z", "running": 0, "size": 1}}] 2022-03-08T23:21:54.245 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:21:55.166 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:21:55.166 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:21:54.776563Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:21:54.776563Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:21:54.776563Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:21:54.776563Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:54.776563Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:54.776563Z", "running": 0, "size": 1}}] 2022-03-08T23:22:05.189 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:05.801 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:05.801 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:05.691283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:05.691283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:05.691283Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:05.691283Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:05.691283Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:05.691283Z", "running": 0, "size": 1}}] 2022-03-08T23:22:15.822 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:16.445 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:16.446 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:16.334123Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:16.334123Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:16.334123Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:16.334123Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:16.334123Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:16.334123Z", "running": 0, "size": 1}}] 2022-03-08T23:22:26.471 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:27.103 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:27.103 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:26.970262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:26.970262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:26.970262Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:26.970262Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:26.970262Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:26.970262Z", "running": 0, "size": 1}}] 2022-03-08T23:22:37.142 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:37.740 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:37.758 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:37.628157Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:37.628157Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:37.628157Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:37.628157Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:37.628157Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:37.628157Z", "running": 0, "size": 1}}] 2022-03-08T23:22:47.766 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:48.361 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:48.362 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:48.250287Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:48.250287Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:48.250287Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:48.250287Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:48.250287Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:48.250287Z", "running": 0, "size": 1}}] 2022-03-08T23:22:58.389 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:22:59.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:22:59.007 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:22:58.897964Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:22:58.897964Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:22:58.897964Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:22:58.897964Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:58.897964Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:58.897964Z", "running": 0, "size": 1}}] 2022-03-08T23:23:09.029 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:23:09.655 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:23:09.655 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:23:09.540894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:23:09.540894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:23:09.540894Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:23:09.540894Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:09.540894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:09.540894Z", "running": 0, "size": 1}}] 2022-03-08T23:23:19.676 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:23:20.294 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:23:20.294 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:23:20.170837Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:23:20.170837Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:23:20.170837Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:23:20.170837Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:20.170837Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:20.170837Z", "running": 0, "size": 1}}] 2022-03-08T23:23:30.316 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:23:30.949 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:23:30.950 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:23:30.831683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:23:30.831683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:23:30.831683Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:23:30.831683Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:30.831683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:30.831683Z", "running": 0, "size": 1}}] 2022-03-08T23:23:40.971 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:23:41.590 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:23:41.591 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:23:41.479161Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:23:41.479161Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:23:41.479161Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:23:41.479161Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:41.479161Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:41.479161Z", "running": 0, "size": 1}}] 2022-03-08T23:23:51.612 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:23:52.213 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:23:52.213 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:23:52.102407Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:23:52.102407Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:23:52.102407Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:23:52.102407Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:52.102407Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:52.102407Z", "running": 0, "size": 1}}] 2022-03-08T23:24:02.239 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:02.864 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:02.865 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:02.751637Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:02.751637Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:02.751637Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:02.751637Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:02.751637Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:02.751637Z", "running": 0, "size": 1}}] 2022-03-08T23:24:12.887 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:13.509 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:13.397934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:13.397934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:13.397934Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:13.397934Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:13.397934Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:13.397934Z", "running": 0, "size": 1}}] 2022-03-08T23:24:23.529 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:24.160 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:24.160 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:24.019462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:24.019462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:24.019462Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:24.019462Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:24.019462Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:24.019462Z", "running": 0, "size": 1}}] 2022-03-08T23:24:34.183 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:34.843 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:34.844 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:34.710053Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:34.710053Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:34.710053Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:34.710053Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:34.710053Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:34.710053Z", "running": 0, "size": 1}}] 2022-03-08T23:24:44.865 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:45.725 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:45.726 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:45.577853Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:45.577853Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:45.577853Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:45.577853Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:45.577853Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:45.577853Z", "running": 0, "size": 1}}] 2022-03-08T23:24:55.746 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:24:56.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:24:56.353 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:24:56.235492Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:24:56.235492Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:24:56.235492Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:24:56.235492Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:56.235492Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:56.235492Z", "running": 0, "size": 1}}] 2022-03-08T23:25:06.379 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:25:06.979 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:25:06.979 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:25:06.870909Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:25:06.870909Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:25:06.870909Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:25:06.870909Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:06.870909Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:06.870909Z", "running": 0, "size": 1}}] 2022-03-08T23:25:17.001 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:25:17.606 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:25:17.606 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:25:17.503832Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:25:17.503832Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:25:17.503832Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:25:17.503832Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:17.503832Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:17.503832Z", "running": 0, "size": 1}}] 2022-03-08T23:25:27.626 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:25:28.236 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:25:28.236 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:25:28.119806Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:25:28.119806Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:25:28.119806Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:25:28.119806Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:28.119806Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:28.119806Z", "running": 0, "size": 1}}] 2022-03-08T23:25:38.261 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:25:38.927 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:25:38.927 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:25:38.812178Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:25:38.812178Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:25:38.812178Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:25:38.812178Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:38.812178Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:38.812178Z", "running": 0, "size": 1}}] 2022-03-08T23:25:48.947 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:25:49.550 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:25:49.551 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:25:49.440428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:25:49.440428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:25:49.440428Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:25:49.440428Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:49.440428Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:49.440428Z", "running": 0, "size": 1}}] 2022-03-08T23:25:59.571 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:00.159 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:00.159 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:00.046163Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:00.046163Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:00.046163Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:00.046163Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:00.046163Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:00.046163Z", "running": 0, "size": 1}}] 2022-03-08T23:26:10.180 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:10.815 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:10.815 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:10.709581Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:10.709581Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:10.709581Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:10.709581Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:10.709581Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:10.709581Z", "running": 0, "size": 1}}] 2022-03-08T23:26:20.840 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:21.481 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:21.481 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:21.354526Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:21.354526Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:21.354526Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:21.354526Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:21.354526Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:21.354526Z", "running": 0, "size": 1}}] 2022-03-08T23:26:31.506 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:32.137 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:32.138 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:32.031935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:32.031935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:32.031935Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:32.031935Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:32.031935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:32.031935Z", "running": 0, "size": 1}}] 2022-03-08T23:26:42.164 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:42.777 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:42.777 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:42.670468Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:42.670468Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:42.670468Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:42.670468Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:42.670468Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:42.670468Z", "running": 0, "size": 1}}] 2022-03-08T23:26:52.799 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:26:53.423 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:26:53.424 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:26:53.321077Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:26:53.321077Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:26:53.321077Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:26:53.321077Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:53.321077Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:53.321077Z", "running": 0, "size": 1}}] 2022-03-08T23:27:03.448 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:04.087 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:04.088 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:03.972078Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:03.972078Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:03.972078Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:03.972078Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:03.972078Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:03.972078Z", "running": 0, "size": 1}}] 2022-03-08T23:27:14.114 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:14.738 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:14.739 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:14.633428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:14.633428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:14.633428Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:14.633428Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:14.633428Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:14.633428Z", "running": 0, "size": 1}}] 2022-03-08T23:27:24.763 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:25.416 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:25.416 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:25.312293Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:25.312293Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:25.312293Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:25.312293Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:25.312293Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:25.312293Z", "running": 0, "size": 1}}] 2022-03-08T23:27:35.437 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:36.225 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:36.226 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:35.937572Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:35.937572Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:35.937572Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:35.937572Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:35.937572Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:35.937572Z", "running": 0, "size": 1}}] 2022-03-08T23:27:46.250 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:46.965 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:46.965 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:46.848401Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:46.848401Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:46.848401Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:46.848401Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:46.848401Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:46.848401Z", "running": 0, "size": 1}}] 2022-03-08T23:27:56.990 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:27:57.590 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:27:57.590 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:27:57.476462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:27:57.476462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:27:57.476462Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:27:57.476462Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:57.476462Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:57.476462Z", "running": 0, "size": 1}}] 2022-03-08T23:28:07.614 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:28:08.281 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:28:08.281 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:28:08.175860Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:28:08.175860Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:28:08.175860Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:28:08.175860Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:08.175860Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:08.175860Z", "running": 0, "size": 1}}] 2022-03-08T23:28:18.305 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:28:18.949 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:28:18.949 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:28:18.827992Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:28:18.827992Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:28:18.827992Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:28:18.827992Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:18.827992Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:18.827992Z", "running": 0, "size": 1}}] 2022-03-08T23:28:28.974 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:28:29.903 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:28:29.903 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:28:29.801341Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:28:29.801341Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:28:29.801341Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:28:29.801341Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:29.801341Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:29.801341Z", "running": 0, "size": 1}}] 2022-03-08T23:28:39.925 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:28:40.567 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:28:40.568 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:28:40.458564Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:28:40.458564Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:28:40.458564Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:28:40.458564Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:40.458564Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:40.458564Z", "running": 0, "size": 1}}] 2022-03-08T23:28:50.588 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:28:51.233 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:28:51.234 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:28:51.120329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:28:51.120329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:28:51.120329Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:28:51.120329Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:51.120329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:51.120329Z", "running": 0, "size": 1}}] 2022-03-08T23:29:01.256 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:01.870 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:01.871 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:01.755263Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:01.755263Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:01.755263Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:01.755263Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:01.755263Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:01.755263Z", "running": 0, "size": 1}}] 2022-03-08T23:29:11.893 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:12.783 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:12.672989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:12.672989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:12.672989Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:12.672989Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:12.672989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:12.672989Z", "running": 0, "size": 1}}] 2022-03-08T23:29:22.806 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:23.437 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:23.438 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:23.331329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:23.331329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:23.331329Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:23.331329Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:23.331329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:23.331329Z", "running": 0, "size": 1}}] 2022-03-08T23:29:33.462 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:34.075 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:34.076 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:33.970006Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:33.970006Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:33.970006Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:33.970006Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:33.970006Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:33.970006Z", "running": 0, "size": 1}}] 2022-03-08T23:29:43.787 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:29:43.772998 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:29:44.112 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:44.422 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:29:44.407381 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:29:44.400+0000 7ff8df827340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:29:44.423 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:29:44.400+0000 7ff8df827340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:29:44.423 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:29:44.400+0000 7ff8df827340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:29:44.423 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:29:44.423 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:29:44.407412 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:29:44.400+0000 7ff8df827340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:29:44.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:29:44.400+0000 7ff8df827340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:29:44.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:29:44.400+0000 7ff8df827340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:29:44.424 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:29:44.763 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:44.763 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:44.614878Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:44.614878Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:44.614878Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:44.614878Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:44.614878Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:44.614878Z", "running": 0, "size": 1}}] 2022-03-08T23:29:54.785 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:29:55.494 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:29:55.494 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:29:55.354150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:29:55.354150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:29:55.354150Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:29:55.354150Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:55.354150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:55.354150Z", "running": 0, "size": 1}}] 2022-03-08T23:30:05.516 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:06.110 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:06.110 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:05.997073Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:05.997073Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:05.997073Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:05.997073Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:05.997073Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:05.997073Z", "running": 0, "size": 1}}] 2022-03-08T23:30:16.134 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:16.780 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:16.781 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:16.644854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:16.644854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:16.644854Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:16.644854Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:16.644854Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:16.644854Z", "running": 0, "size": 1}}] 2022-03-08T23:30:26.807 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:27.509 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:27.510 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:27.388628Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:27.388628Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:27.388628Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:27.388628Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:27.388628Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:27.388628Z", "running": 0, "size": 1}}] 2022-03-08T23:30:37.532 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:38.147 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:38.147 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:38.000296Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:38.000296Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:38.000296Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:38.000296Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:38.000296Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:38.000296Z", "running": 0, "size": 1}}] 2022-03-08T23:30:48.171 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:48.821 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:48.821 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:48.691283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:48.691283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:48.691283Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:48.691283Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:48.691283Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:48.691283Z", "running": 0, "size": 1}}] 2022-03-08T23:30:58.846 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:30:59.694 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:30:59.694 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:30:59.566337Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:30:59.566337Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:30:59.566337Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:30:59.566337Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:59.566337Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:59.566337Z", "running": 0, "size": 1}}] 2022-03-08T23:31:09.716 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:31:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:31:12.788 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:31:12.678976Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:31:12.678976Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:31:12.678976Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:31:12.678976Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:12.678976Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:12.678976Z", "running": 0, "size": 1}}] 2022-03-08T23:31:22.809 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:31:23.497 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:31:23.497 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:31:23.351753Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:31:23.351753Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:31:23.351753Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:31:23.351753Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:23.351753Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:23.351753Z", "running": 0, "size": 1}}] 2022-03-08T23:31:33.521 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:31:34.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:31:34.188 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:31:34.075558Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:31:34.075558Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:31:34.075558Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:31:34.075558Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:34.075558Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:34.075558Z", "running": 0, "size": 1}}] 2022-03-08T23:31:44.212 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:31:44.848 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:31:44.848 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:31:44.738118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:31:44.738118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:31:44.738118Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:31:44.738118Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:44.738118Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:44.738118Z", "running": 0, "size": 1}}] 2022-03-08T23:31:54.872 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:31:55.523 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:31:55.523 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:31:55.408877Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:31:55.408877Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:31:55.408877Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:31:55.408877Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:55.408877Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:55.408877Z", "running": 0, "size": 1}}] 2022-03-08T23:32:05.551 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:06.206 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:06.207 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:06.056014Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:06.056014Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:06.056014Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:06.056014Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:06.056014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:06.056014Z", "running": 0, "size": 1}}] 2022-03-08T23:32:16.231 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:16.834 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:16.835 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:16.733538Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:16.733538Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:16.733538Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:16.733538Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:16.733538Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:16.733538Z", "running": 0, "size": 1}}] 2022-03-08T23:32:26.860 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:27.550 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:27.550 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:27.436685Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:27.436685Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:27.436685Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:27.436685Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:27.436685Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:27.436685Z", "running": 0, "size": 1}}] 2022-03-08T23:32:37.576 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:38.212 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:38.212 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:38.100151Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:38.100151Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:38.100151Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:38.100151Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:38.100151Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:38.100151Z", "running": 0, "size": 1}}] 2022-03-08T23:32:48.236 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:48.861 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:48.862 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:48.748325Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:48.748325Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:48.748325Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:48.748325Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:48.748325Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:48.748325Z", "running": 0, "size": 1}}] 2022-03-08T23:32:58.891 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:32:59.632 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:32:59.632 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:32:59.449631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:32:59.449631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:32:59.449631Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:32:59.449631Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:32:59.449631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:32:59.449631Z", "running": 0, "size": 1}}] 2022-03-08T23:33:09.657 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:33:10.282 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:33:10.282 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:33:10.161068Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:33:10.161068Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:33:10.161068Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:33:10.161068Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:33:10.161068Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:33:10.161068Z", "running": 0, "size": 1}}] 2022-03-08T23:33:20.304 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:33:20.879 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:33:20.879 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:33:20.773844Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:33:20.773844Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:33:20.773844Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:33:20.773844Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:33:20.773844Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:33:20.773844Z", "running": 0, "size": 1}}] 2022-03-08T23:33:30.903 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:33:31.583 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:33:31.583 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:33:31.450832Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:33:31.450832Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:33:31.450832Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:33:31.450832Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:33:31.450832Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:33:31.450832Z", "running": 0, "size": 1}}] 2022-03-08T23:33:41.606 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:33:42.170 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:33:42.171 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:33:42.059689Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:33:42.059689Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:33:42.059689Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:33:42.059689Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:33:42.059689Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:33:42.059689Z", "running": 0, "size": 1}}] 2022-03-08T23:33:52.194 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:33:52.792 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:33:52.792 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:33:52.686834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:33:52.686834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:33:52.686834Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:33:52.686834Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:33:52.686834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:33:52.686834Z", "running": 0, "size": 1}}] 2022-03-08T23:34:02.817 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:34:03.500 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:34:03.500 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:34:03.385845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:34:03.385845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:34:03.385845Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:34:03.385845Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:34:03.385845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:34:03.385845Z", "running": 0, "size": 1}}] 2022-03-08T23:34:13.525 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:34:14.159 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:34:14.159 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:34:14.013866Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:34:14.013866Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:34:14.013866Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:34:14.013866Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:34:14.013866Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:34:14.013866Z", "running": 0, "size": 1}}] 2022-03-08T23:34:24.181 DEBUG:teuthology.orchestra.run.smithi117:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-5xdk6 -- ceph orch ls -f json 2022-03-08T23:34:24.838 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-03-08T23:34:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:18:48.000000Z", "last_refresh": "2022-03-08T23:34:24.723298Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:42.000000Z", "last_refresh": "2022-03-08T23:34:24.723298Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:01:16.000000Z", "last_refresh": "2022-03-08T23:34:24.723298Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:23.000000Z", "last_refresh": "2022-03-08T23:34:24.723298Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:34:24.723298Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:34:24.723298Z", "running": 0, "size": 1}}] 2022-03-08T23:34:24.863 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:34:24.864 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:34:24.865 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:34:24.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:34:25.028 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:34:25.029 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:34:25.068 DEBUG:teuthology.orchestra.remote:smithi117:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T23:34:25.080 DEBUG:teuthology.orchestra.run.smithi117:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T23:34:25.144 INFO:teuthology.orchestra.run.smithi117.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:34:25.151 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:34:25.151 DEBUG:teuthology.orchestra.run.smithi117:> kubectl delete -f cluster.yaml 2022-03-08T23:34:25.284 INFO:teuthology.orchestra.run.smithi117.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:34:25.286 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:25.270015 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:34:25.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:25.860442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:25.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:25.875007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:25.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:25.875041 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:36.136 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:36.119906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:36.148 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:36.132105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:46.394 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:46.378367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:46.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:46.390455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:56.664 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:56.648999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:56.676 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:34:56.660636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:06.893 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:06.878458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:06.906 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:06.891350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:17.126 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:17.111854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:17.138 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:17.123594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:27.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:27.353737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:27.380 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:27.366261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:37.603 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:37.588703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:37.615 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:37.600530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:47.835 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:47.821172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:47.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:47.834348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:58.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:58.063343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:58.090 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:35:58.076438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:08.312 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:08.298784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:08.328 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:08.314576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:18.551 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:18.537620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:18.563 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:18.550236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:28.777 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:28.763730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:28.830 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:28.816706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:39.078 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:39.061843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:39.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:39.076631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:49.349 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:49.336457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:49.372 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:49.359595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:59.588 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:59.575336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:59.598 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:36:59.586069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:09.832 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:09.819879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:09.845 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:09.832767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:20.069 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:20.056333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:20.082 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:20.069731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:30.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:30.291874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:30.316 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:30.304036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:40.540 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:40.527531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:40.550 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:40.537997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:50.764 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:50.752183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:50.777 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:37:50.765109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:00.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:00.983655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:01.008 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:00.996255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:11.250 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:11.237839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:11.263 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:11.251659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:21.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:21.515078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:21.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:21.546747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:31.776 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:31.764404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:31.788 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:31.776991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:42.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:41.998656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:42.022 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:42.010823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:52.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:52.233983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:52.259 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:38:52.247296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:02.573 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:02.561685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:02.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:02.573981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:12.847 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:12.835750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:12.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:12.848670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:23.082 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:23.070466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:23.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:23.085000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:33.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:33.305726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:33.341 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:33.325950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:43.591 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:43.579203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:43.603 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:43.591815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:53.832 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:53.820573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:53.844 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:39:53.833047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:04.063 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:04.051853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:04.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:04.065981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:14.295 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:14.283788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:14.309 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:14.298027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:24.530 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:24.518446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:24.541 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:24.529537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:34.780 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:34.768260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:34.793 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:34.781703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:45.047 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:45.035359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:45.065 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:45.054073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:55.488 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:55.476065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:55.520 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:40:55.508347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:05.784 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:05.772839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:05.813 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:05.802160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:16.027 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:16.015242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:16.038 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:16.026460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:26.263 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:26.251548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:26.273 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:26.261699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:36.516 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:36.503926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:36.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:36.514981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:46.763 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:46.751342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:46.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:46.760391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:56.997 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:56.984755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:57.008 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:41:56.996575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:07.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:07.218078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:07.242 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:07.230332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:17.462 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:17.449955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:17.477 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:17.465545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:27.698 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:27.685624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:27.711 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:27.699252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:37.932 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:37.919461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:37.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:37.930954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:48.191 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:48.178842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:48.203 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:48.191089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:58.426 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:58.413986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:58.435 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:42:58.423111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:08.663 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:08.650673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:08.673 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:08.661932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:18.909 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:18.897793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:18.927 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:18.915595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:29.165 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:29.154176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:29.177 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:29.165759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:39.420 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:39.409942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:39.430 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:39.419324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:49.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:49.638731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:49.661 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:49.651111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:59.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:59.871306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:59.892 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:43:59.881728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:10.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:10.104060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:10.123 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:10.112976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:20.341 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:20.330523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:20.352 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:20.342720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:30.620 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:30.610575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:30.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:30.618943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:40.847 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:40.837670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:40.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:40.850849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:51.080 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:51.069973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:51.090 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:44:51.081105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:01.319 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:01.310163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:01.330 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:01.320767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:11.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:11.535302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:11.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:11.547214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:21.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:21.762862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:21.782 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:21.773197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:32.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:32.001079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:32.023 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:32.014390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:42.248 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:42.239533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:42.260 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:42.251071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:52.522 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:52.513258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:52.537 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:45:52.526528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:02.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:02.744380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:02.765 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:02.757191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:12.986 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:12.977171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:13.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:13.033659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:23.262 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:23.253867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:23.275 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:23.267228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:24.423 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:24.414372 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-08T23:46:25.041 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:25.031918 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:46:25.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:46:25.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:46:25.042 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:46:25.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:25.031951 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:46:25.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:46:25.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08T23:46:25.022+0000 7fc2b1d25340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:46:25.043 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:46:33.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:33.489090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:33.509 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:33.500716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:43.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:43.737955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:43.758 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:43.750323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:53.983 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:53.974496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:53.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:46:53.987462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:04.218 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:04.210494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:04.228 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:04.220073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:14.447 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:14.438766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:14.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:14.451091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:24.684 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:24.675531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:24.697 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:24.689328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:34.920 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:34.911958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:34.936 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:34.926896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:45.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:45.166100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:45.184 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:45.175919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:55.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:55.397741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:55.421 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:47:55.413485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:05.640 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:05.632456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:05.710 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:05.701738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:15.931 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:15.922993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:15.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:15.935265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:26.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:26.153454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:26.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:26.167170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:36.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:36.387275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:36.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:36.398292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:46.627 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:46.618882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:46.636 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:46.628899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:56.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:56.846500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:56.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:48:56.864357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:07.125 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:07.105570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:07.125 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:07.117440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:17.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:17.337393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:17.359 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:17.349204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:27.579 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:27.570732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:27.591 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:27.583316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:37.812 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:37.803953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:37.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:37.813935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:48.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:48.033701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:48.053 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:48.045207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:58.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:58.271805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:58.293 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:49:58.285538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:08.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:08.506283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:08.562 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:08.553266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:18.789 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:18.780815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:18.801 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:18.793413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:29.058 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:29.050038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:29.068 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:29.059895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:39.293 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:39.284871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:39.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:39.298443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:49.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:49.519237 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:50:49.512808148 +0000 UTC m=+3055.332169588 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:50:49.555 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:49.547259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:49.563 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:49.555310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:59.785 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:59.776617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:59.793 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:50:59.785732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:10.015 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:10.007640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:10.028 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:10.020343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:20.252 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:20.244125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:20.265 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:20.256343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:30.485 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:30.476482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:30.496 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:30.488828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:40.738 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:40.730794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:40.752 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:40.745276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:50.974 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:50.965891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:50.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:51:50.980973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:01.210 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:01.202981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:01.222 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:01.215482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:11.447 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:11.439952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:11.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:11.476790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:21.721 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:21.714600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:21.767 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:21.760134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:31.982 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:31.974938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:31.993 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:31.986653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:42.214 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:42.207069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:42.288 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:42.281425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:52.506 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:52.499468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:52.517 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:52:52.510439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:02.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:02.728535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:02.749 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:02.742639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:12.971 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:12.963983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:12.985 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:12.978069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:23.206 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:23.198840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:23.218 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:23.211843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:33.439 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:33.431997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:33.451 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:33.443254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:43.674 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:43.667558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:43.685 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:43.678877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:53.909 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:53.902373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:53.920 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:53:53.913824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:04.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:04.134869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:04.155 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:04.147821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:14.373 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:14.366899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:14.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:14.378292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:24.612 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:24.605343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:24.620 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:24.613774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:34.845 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:34.837952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:34.875 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:34.868956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:45.120 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:45.113988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:45.132 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:45.126238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:55.354 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:55.347971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:55.367 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:54:55.361222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:05.586 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:05.579945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:05.602 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:05.595125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:15.824 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:15.817581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:15.834 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:15.828094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:26.060 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:26.053047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:26.071 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:26.065011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:36.296 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:36.289396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:36.309 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:36.303411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:46.562 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:46.555717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:46.572 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:46.565761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:56.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:56.785660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:56.806 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:55:56.800144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:07.029 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:07.022116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:07.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:07.036439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:17.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:17.259800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:17.274 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:17.268320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:27.510 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:27.504333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:27.542 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:27.536162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:37.759 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:37.752772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:37.769 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:37.763236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:47.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:47.981734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:47.998 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:47.992091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:58.222 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:58.215684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:58.233 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:56:58.226804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:08.448 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:08.442089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:08.458 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:08.451145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:18.677 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:18.670954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:18.689 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:18.682731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:28.914 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:28.907527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:28.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:28.918907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:39.146 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:39.140172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:39.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:39.150357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:49.378 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:49.372181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:49.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:49.384974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:59.616 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:59.609872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:59.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:57:59.623236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:09.853 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:09.846170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:09.866 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:09.859934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:20.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:20.082669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:20.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:20.096158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:30.327 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:30.320898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:30.340 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:30.334032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:40.562 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:40.555672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:40.572 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:40.566136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:50.815 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:50.808970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:50.828 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:58:50.821611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:01.047 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:01.040378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:01.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:01.075534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:11.315 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:11.308404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:11.325 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:11.318977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:21.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:21.537834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:21.553 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:21.547005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:31.823 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:31.816653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:31.835 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:31.828415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:42.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:42.049640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:42.070 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:42.063960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:52.296 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:52.288225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:52.306 INFO:tasks.rook.operator.smithi117.stdout:2022-03-08 23:59:52.300364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:02.528 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:02.521194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:02.537 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:02.531228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:12.780 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:12.773103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:12.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:12.785763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:23.012 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:23.005489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:23.040 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:23.033852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:33.270 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:33.263221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:33.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:33.274928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:43.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:43.507309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:43.525 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:43.518717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:53.765 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:53.758311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:53.775 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:00:53.768962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:03.990 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:03.983144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:03.999 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:03.993004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:14.222 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:14.215576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:14.234 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:14.228044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:24.455 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:24.448295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:24.479 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:24.462320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:34.693 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:34.686659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:34.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:34.735003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:44.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:44.955507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:44.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:44.966683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:55.194 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:55.187180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:55.210 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:01:55.203645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:05.433 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:05.425845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:05.447 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:05.440361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:15.668 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:15.661721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:15.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:15.673371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:25.908 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:25.901054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:25.920 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:25.913709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:36.144 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:36.137140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:36.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:36.149668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:46.383 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:46.376598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:46.398 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:46.391378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:56.649 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:56.642285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:56.660 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:02:56.654407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:05.052 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:05.039062 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T00:03:05.699 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:05.691473 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:03:05.684+0000 7f2722140340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:03:05.699 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:03:05.684+0000 7f2722140340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:03:05.699 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:03:05.684+0000 7f2722140340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:03:05.699 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:03:05.700 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:05.691506 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:03:05.684+0000 7f2722140340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:03:05.700 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:03:05.684+0000 7f2722140340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:03:05.700 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:03:05.684+0000 7f2722140340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:03:05.700 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:03:06.936 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:06.929561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:06.948 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:06.941240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:17.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:17.164410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:17.184 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:17.177579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:27.408 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:27.401895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:27.420 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:27.413927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:37.683 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:37.677118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:37.696 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:37.689636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:47.912 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:47.905830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:47.924 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:47.917083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:58.154 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:58.147518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:58.166 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:03:58.160022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:08.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:08.381475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:08.399 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:08.392883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:18.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:18.612049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:18.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:18.622948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:28.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:28.843782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:28.863 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:28.856736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:39.096 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:39.090299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:39.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:39.150219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:49.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:49.386088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:49.402 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:49.396066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:59.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:59.615809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:59.635 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:04:59.628538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:09.880 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:09.874063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:09.893 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:09.886804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:20.150 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:20.143479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:20.163 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:20.156553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:30.412 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:30.404998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:30.460 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:30.454207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:40.730 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:40.724054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:40.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:40.736027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:50.959 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:50.952971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:50.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:05:50.967126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:01.192 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:01.185987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:01.202 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:01.196033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:11.422 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:11.416605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:11.430 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:11.424909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:21.656 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:21.649762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:21.667 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:21.661649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:31.900 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:31.894267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:31.912 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:31.906091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:42.162 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:42.156226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:42.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:42.169608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:52.398 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:52.392253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:52.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:06:52.404701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:02.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:02.622973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:02.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:02.633552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:12.859 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:12.853086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:12.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:12.867235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:23.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:23.090698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:23.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:23.104841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:33.331 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:33.325558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:33.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:33.338440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:43.561 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:43.554889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:43.574 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:43.568752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:53.790 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:53.784206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:53.801 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:07:53.795588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:04.023 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:04.016840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:04.032 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:04.026609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:04.321 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:04.314928 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:08:04.306073686 +0000 UTC m=+4090.125435078 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:08:14.262 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:14.256049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:14.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:14.263706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:24.490 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:24.484223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:24.502 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:24.494919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:34.717 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:34.710739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:34.728 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:34.722257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:44.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:44.943101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:44.967 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:44.958144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:55.194 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:55.188556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:55.205 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:08:55.200054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:05.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:05.241051 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:09:05.224519864 +0000 UTC m=+4151.043881260 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:09:05.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:05.417708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:05.436 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:05.430705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:15.658 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:15.651694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:15.670 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:15.664698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:25.895 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:25.888597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:25.907 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:25.900805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:36.130 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:36.123863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:36.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:36.136775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:46.364 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:46.357799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:46.374 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:46.368249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:56.596 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:56.590205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:56.608 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:09:56.602783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:06.830 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:06.824273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:06.841 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:06.835636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:17.063 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:17.057247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:17.076 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:17.070162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:27.297 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:27.291068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:27.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:27.298845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:37.561 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:37.554831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:37.574 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:37.567896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:47.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:47.785952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:47.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:47.798243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:58.027 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:58.021129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:58.039 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:10:58.033028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:08.290 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:08.283831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:08.301 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:08.295600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:18.524 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:18.518012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:18.538 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:18.532534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:28.761 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:28.754487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:28.773 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:28.767416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:38.997 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:38.990962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:39.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:39.003861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:49.231 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:49.224982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:49.243 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:49.237532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:59.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:59.476602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:59.496 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:11:59.489631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:09.716 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:09.709235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:09.730 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:09.723544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:19.950 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:19.943724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:19.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:19.956568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:30.213 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:30.206277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:30.225 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:30.219020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:40.444 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:40.436775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:40.455 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:40.448399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:50.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:50.665903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:50.748 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:12:50.741778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:00.972 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:00.965348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:00.981 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:00.974719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:11.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:11.194363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:11.213 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:11.206621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:21.433 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:21.426314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:21.447 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:21.439288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:31.664 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:31.657606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:31.676 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:31.669772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:41.899 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:41.892463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:41.906 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:41.900361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:52.130 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:52.123773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:52.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:13:52.135174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:02.370 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:02.363466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:02.383 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:02.376085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:12.605 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:12.598756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:12.620 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:12.613723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:22.843 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:22.836367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:22.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:22.848593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:33.082 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:33.075887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:33.091 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:33.083922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:43.311 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:43.304681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:43.320 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:43.313927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:53.543 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:53.536504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:53.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:14:53.548870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:03.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:03.764970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:03.783 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:03.776555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:13.996 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:13.989495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:14.025 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:14.017607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:24.241 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:24.234387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:24.249 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:24.242383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:34.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:34.495660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:34.520 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:34.512647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:44.740 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:44.732505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:44.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:44.784626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:55.011 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:55.003908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:55.022 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:15:55.015125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:05.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:05.238420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:05.258 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:05.251398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:15.482 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:15.474236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:15.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:15.488457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:25.718 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:25.710285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:25.734 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:25.727233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:36.002 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:35.995185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:36.015 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:36.007821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:46.271 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:46.263593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:46.289 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:46.281826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:56.507 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:56.499275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:56.520 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:16:56.512880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:06.741 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:06.733382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:06.754 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:06.746910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:16.989 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:16.981476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:17.003 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:16.995571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:27.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:27.246232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:27.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:27.257378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:37.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:37.475470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:37.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:37.487331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:47.717 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:47.710040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:47.726 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:47.718553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:57.944 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:57.936345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:57.955 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:17:57.947605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:08.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:08.166696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:08.185 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:08.178102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:18.408 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:18.400325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:18.420 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:18.412643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:28.641 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:28.633020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:28.653 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:28.645222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:38.875 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:38.867451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:38.887 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:38.878665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:49.107 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:49.099233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:49.121 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:49.113441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:59.346 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:59.338323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:59.366 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:18:59.357570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:09.584 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:09.571577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:09.588 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:09.580011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:19.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:19.868510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:19.890 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:19.882042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:30.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:30.101554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:30.121 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:30.113552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:40.353 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:40.344634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:40.362 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:40.353570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:45.708 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:45.699446 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T00:19:46.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:46.358534 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:19:46.349+0000 7f668777f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:19:46.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:19:46.349+0000 7f668777f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:19:46.369 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:19:46.349+0000 7f668777f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:19:46.369 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:19:46.369 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:46.358564 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:19:46.349+0000 7f668777f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:19:46.369 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:19:46.349+0000 7f668777f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:19:46.369 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:19:46.349+0000 7f668777f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:19:46.370 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:19:50.604 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:50.596013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:50.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:19:50.605758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:00.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:00.828501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:00.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:00.839379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:11.083 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:11.074912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:11.095 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:11.087260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:21.475 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:21.466799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:21.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:21.475968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:31.729 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:31.720707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:31.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:31.732833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:41.986 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:41.977676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:41.998 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:41.989615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:52.216 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:52.207571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:52.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:20:52.221567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:02.449 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:02.439872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:02.461 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:02.452729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:12.683 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:12.674020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:12.695 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:12.686448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:22.915 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:22.906282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:22.929 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:22.919930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:33.153 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:33.144190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:33.169 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:33.160370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:43.386 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:43.377352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:43.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:43.385893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:53.613 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:53.604168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:53.627 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:21:53.618309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:03.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:03.839203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:03.861 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:03.851673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:14.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:14.071942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:14.095 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:14.084739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:24.311 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:24.301794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:24.323 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:24.313906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:34.547 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:34.536698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:34.559 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:34.547800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:44.779 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:44.769426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:44.799 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:44.789942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:55.027 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:55.015644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:55.037 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:22:55.027557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:05.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:05.256406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:05.279 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:05.269935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:15.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:15.486464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:15.508 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:15.498553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:25.730 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:25.720077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:25.741 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:25.732056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:35.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:35.952826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:35.976 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:35.966964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:46.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:46.191260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:46.212 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:46.202903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:56.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:56.421110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:56.445 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:23:56.435608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:06.697 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:06.687238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:06.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:06.725232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:16.956 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:16.946071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:16.971 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:16.958179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:27.243 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:27.233538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:27.252 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:27.243197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:37.525 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:37.515660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:37.537 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:37.527889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:47.761 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:47.751383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:47.773 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:47.764301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:57.993 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:57.983942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:58.007 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:24:57.997218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:08.227 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:08.217613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:08.239 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:08.229688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:18.460 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:18.451016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:18.473 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:18.464428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:28.694 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:28.684696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:28.706 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:28.696486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:38.928 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:38.919204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:38.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:38.933971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:49.166 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:49.156391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:49.177 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:49.167738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:59.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:59.390586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:59.412 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:25:59.402922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:09.634 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:09.624578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:09.686 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:09.677464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:19.937 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:19.927305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:19.947 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:19.938415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:20.219 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:20.210239 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:26:20.198747303 +0000 UTC m=+5186.018108745 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:26:30.197 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:30.187653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:30.211 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:30.200579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:40.452 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:40.442247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:40.464 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:40.454366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:50.712 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:50.702601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:50.723 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:26:50.714494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:00.951 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:00.941219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:00.964 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:00.955462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:11.183 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:11.173576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:11.196 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:11.187616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:21.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:21.406799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:21.428 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:21.418934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:31.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:31.637390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:31.657 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:31.648455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:41.905 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:41.896282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:41.919 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:41.910652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:52.143 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:52.133887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:52.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:27:52.146640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:02.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:02.383505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:02.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:02.396432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:12.631 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:12.622333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:12.640 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:12.631269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:22.898 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:22.888962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:22.910 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:22.901149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:33.126 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:33.116366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:33.144 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:33.135041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:43.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:43.382970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:43.410 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:43.401050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:53.673 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:53.663913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:53.681 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:28:53.672485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:03.904 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:03.894814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:03.916 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:03.907433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:14.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:14.131551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:14.152 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:14.143181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:24.373 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:24.363634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:24.426 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:24.417534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:34.689 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:34.679943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:34.701 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:34.692091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:44.926 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:44.916673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:44.944 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:44.930999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:55.157 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:55.147515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:55.170 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:29:55.161738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:05.430 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:05.421109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:05.444 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:05.433190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:15.659 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:15.649820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:15.675 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:15.665785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:25.894 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:25.884712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:25.905 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:25.896719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:36.163 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:36.154669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:36.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:36.165039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:46.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:46.386515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:46.410 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:46.401220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:56.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:56.620400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:56.643 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:30:56.634372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:06.864 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:06.855368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:06.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:06.868006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:17.099 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:17.089749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:17.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:17.106220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:27.337 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:27.327598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:27.350 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:27.341540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:37.571 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:37.561606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:37.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:37.576531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:47.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:47.795625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:47.818 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:47.809005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:58.038 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:58.028482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:58.050 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:31:58.041253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:08.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:08.268386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:08.287 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:08.278681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:18.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:18.517370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:18.537 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:18.527899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:28.768 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:28.758506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:28.781 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:28.771912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:39.048 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:39.038873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:39.061 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:39.052144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:49.287 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:49.277073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:49.299 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:49.289741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:59.512 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:59.503506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:59.521 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:32:59.511910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:09.741 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:09.732153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:09.752 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:09.743427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:19.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:19.963946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:19.985 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:19.975846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:30.211 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:30.201545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:30.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:30.221426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:40.453 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:40.444280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:40.465 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:40.456084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:50.684 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:50.675399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:50.695 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:33:50.686750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:00.921 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:00.911435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:00.931 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:00.922388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:11.155 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:11.145948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:11.169 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:11.158981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:21.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:21.378807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:21.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:21.391188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:31.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:31.616258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:31.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:31.629728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:31.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:31.629768 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:41.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:41.868297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:41.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:41.882313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:52.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:52.132725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:52.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:34:52.147392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:02.379 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:02.369813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:02.390 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:02.380739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:12.609 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:12.599610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:12.623 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:12.614222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:22.843 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:22.833962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:22.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:22.846357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:33.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:33.072116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:33.096 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:33.085098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:43.315 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:43.306489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:43.326 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:43.317557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:53.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:53.544823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:53.566 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:35:53.557258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:03.785 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:03.776024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:03.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:03.807982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:14.072 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:14.063008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:14.090 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:14.080854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:24.350 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:24.341201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:24.362 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:24.353263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:26.380 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:26.371263 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T00:36:26.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:26.958496 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:36:26.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:36:26.969 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:36:26.969 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:36:26.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:26.958523 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:36:26.970 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:36:26.971 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:36:26.952+0000 7fe0ccc79340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:36:26.971 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:36:34.592 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:34.583341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:34.619 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:34.610341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:44.833 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:44.824052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:44.845 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:44.836036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:55.062 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:55.053621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:55.074 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:36:55.065312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:05.302 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:05.293167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:05.312 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:05.303738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:15.528 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:15.519108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:15.537 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:15.527955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:25.782 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:25.773051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:25.795 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:25.786996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:36.054 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:36.044949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:36.067 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:36.058735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:46.329 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:46.319802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:46.342 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:46.333155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:56.563 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:56.554209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:56.572 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:37:56.563381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:06.793 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:06.784104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:06.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:06.795228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:17.046 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:17.036682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:17.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:17.047290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:27.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:27.271969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:27.293 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:27.284095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:37.513 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:37.504302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:37.560 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:37.551415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:47.778 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:47.769427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:47.791 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:47.782449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:58.013 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:58.003809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:58.024 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:38:58.015769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:08.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:08.236756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:08.259 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:08.250885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:18.479 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:18.470163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:18.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:18.482240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:28.712 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:28.703614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:28.723 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:28.714902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:38.978 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:38.969521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:38.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:38.979043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:49.209 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:49.199755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:49.222 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:49.213325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:59.447 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:59.438058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:59.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:39:59.450516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:09.677 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:09.667913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:09.714 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:09.705931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:19.943 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:19.933946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:19.957 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:19.948035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:30.176 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:30.167248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:30.207 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:30.197759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:40.444 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:40.434644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:40.455 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:40.446974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:50.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:50.662450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:50.686 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:40:50.677273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:00.906 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:00.896946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:00.917 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:00.908751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:11.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:11.132154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:11.157 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:11.148048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:21.379 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:21.370044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:21.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:21.383506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:31.610 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:31.601576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:31.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:31.614001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:41.846 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:41.837628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:41.854 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:41.845665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:52.076 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:52.067300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:52.088 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:41:52.079437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:02.305 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:02.296281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:02.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:02.307976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:12.568 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:12.558836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:12.581 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:12.572198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:22.800 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:22.791391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:22.811 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:22.802645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:33.039 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:33.028703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:33.054 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:33.042984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:43.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:43.260373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:43.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:43.271072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:53.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:53.534761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:53.557 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:42:53.548162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:03.795 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:03.786541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:03.807 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:03.798189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:14.027 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:14.017666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:14.037 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:14.028970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:24.255 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:24.246444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:24.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:24.257676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:34.487 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:34.477652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:34.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:34.488437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:44.718 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:44.708270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:44.729 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:44.720726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:54.960 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:54.951252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:54.971 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:43:54.962496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:05.188 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:05.178357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:05.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:05.191904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:15.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:15.422053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:15.469 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:15.459890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:25.703 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:25.694153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:25.715 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:25.706441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:35.915 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:35.905989 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:44:35.896219308 +0000 UTC m=+6281.715580715 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:44:35.942 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:35.933603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:35.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:35.945579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:46.198 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:46.188754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:46.210 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:46.200314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:56.493 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:56.483929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:56.507 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:44:56.497502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:06.725 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:06.715589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:06.734 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:06.725571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:16.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:16.945002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:16.966 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:16.956083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:27.183 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:27.174047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:27.197 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:27.186995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:37.418 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:37.409402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:37.430 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:37.421243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:47.658 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:47.648327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:47.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:47.663009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:57.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:57.944768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:57.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:45:57.958516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:08.190 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:08.180295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:08.210 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:08.199076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:18.429 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:18.420024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:18.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:18.432047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:28.663 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:28.653727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:28.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:28.663192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:39.036 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:39.026979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:39.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:39.034969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:49.312 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:49.302982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:49.326 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:49.317499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:59.588 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:59.577722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:59.599 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:46:59.590116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:09.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:09.810602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:09.833 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:09.823622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:20.051 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:20.041527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:20.062 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:20.052799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:30.282 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:30.272159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:30.294 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:30.284658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:40.518 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:40.508206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:40.530 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:40.520207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:50.770 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:50.760289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:50.779 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:47:50.769806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:01.011 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:01.001701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:01.022 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:01.013038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:11.243 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:11.233551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:11.255 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:11.245777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:21.472 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:21.462496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:21.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:21.474675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:31.832 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:31.822192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:31.842 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:31.833220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:42.168 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:42.158271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:42.178 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:42.169213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:52.403 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:52.393534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:52.415 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:48:52.405258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:02.723 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:02.712960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:02.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:02.724883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:12.955 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:12.944992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:12.966 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:12.956535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:23.189 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:23.179373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:23.203 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:23.193839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:33.428 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:33.417906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:33.440 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:33.430496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:43.666 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:43.655962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:43.678 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:43.668344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:53.899 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:53.889546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:53.912 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:49:53.901904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:04.132 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:04.121886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:04.140 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:04.130765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:14.360 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:14.349795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:14.374 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:14.364655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:24.595 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:24.585216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:24.605 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:24.595679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:34.853 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:34.843217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:34.869 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:34.859209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:45.084 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:45.073959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:45.147 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:45.136499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:55.363 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:55.353371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:55.375 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:50:55.364838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:05.601 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:05.590874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:05.611 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:05.600892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:15.828 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:15.817777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:15.840 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:15.830817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:26.058 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:26.047602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:26.069 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:26.059449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:36.289 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:36.279354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:36.300 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:36.290515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:46.519 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:46.509162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:46.532 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:46.521670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:56.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:56.743014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:56.764 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:51:56.754375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:06.992 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:06.982083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:07.004 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:06.994428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:17.231 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:17.220663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:17.243 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:17.233683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:27.463 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:27.452848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:27.531 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:27.520978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:37.750 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:37.739014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:37.762 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:37.751898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:48.006 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:47.995907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:48.019 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:48.008937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:58.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:58.258548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:58.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:52:58.271377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:06.976 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:06.965909 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T00:53:07.621 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:07.609937 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:53:07.598+0000 7fca1a597340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:53:07.621 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:53:07.598+0000 7fca1a597340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:53:07.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:53:07.598+0000 7fca1a597340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:53:07.622 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:53:07.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:07.609959 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:53:07.598+0000 7fca1a597340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:53:07.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:53:07.598+0000 7fca1a597340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:53:07.623 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T00:53:07.598+0000 7fca1a597340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:53:07.623 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:53:08.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:08.487949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:08.582 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:08.571729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:18.806 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:18.795119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:18.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:18.808509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:29.040 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:29.029876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:29.048 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:29.038414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:39.270 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:39.259427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:39.283 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:39.272832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:49.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:49.493210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:49.516 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:49.506009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:59.745 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:59.734158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:59.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:53:59.745639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:09.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:09.962588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:09.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:09.976398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:20.206 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:20.195183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:20.216 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:20.205168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:30.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:30.472744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:30.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:30.484864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:40.748 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:40.737129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:40.783 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:40.772658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:51.029 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:51.017563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:51.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:54:51.031761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:01.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:01.253277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:01.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:01.266184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:11.500 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:11.489079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:11.511 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:11.500783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:21.724 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:21.713048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:21.734 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:21.723858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:31.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:31.943095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:31.969 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:31.957954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:42.196 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:42.185082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:42.209 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:42.198748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:52.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:52.420141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:52.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:55:52.430165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:02.696 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:02.684606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:02.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:02.709579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:12.950 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:12.938994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:12.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:12.950805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:23.183 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:23.171410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:23.192 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:23.180794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:33.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:33.399822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:33.422 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:33.411058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:43.641 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:43.630315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:43.654 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:43.643122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:53.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:53.860519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:53.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:56:53.873606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:04.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:04.090928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:04.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:04.104034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:14.335 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:14.323708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:14.348 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:14.337257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:24.575 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:24.563151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:24.587 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:24.576262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:34.809 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:34.797539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:34.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:34.809234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:45.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:45.032245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:45.055 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:45.042818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:55.273 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:55.261615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:55.285 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:57:55.273836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:05.506 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:05.494682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:05.518 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:05.506496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:15.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:15.730589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:15.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:15.741190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:25.975 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:25.962961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:25.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:25.975500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:36.214 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:36.202592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:36.225 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:36.213630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:46.451 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:46.439820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:46.463 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:46.451489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:56.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:56.667553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:56.692 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:58:56.680690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:06.912 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:06.899891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:06.923 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:06.910758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:17.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:17.129216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:17.155 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:17.140527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:27.370 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:27.358031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:27.378 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:27.366840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:37.599 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:37.587207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:37.611 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:37.598934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:47.843 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:47.831001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:47.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:47.842949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:58.074 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:58.062357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:58.111 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 00:59:58.099579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:08.331 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:08.319449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:08.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:08.331642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:18.567 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:18.554692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:18.578 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:18.566366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:28.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:28.793417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:28.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:28.808386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:39.038 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:39.025854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:39.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:39.037429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:49.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:49.253635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:49.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:49.265044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:59.503 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:59.491385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:59.511 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:00:59.499503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:09.730 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:09.718032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:09.742 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:09.730370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:19.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:19.950638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:20.012 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:19.964900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:30.201 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:30.189015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:30.218 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:30.204676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:40.439 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:40.426720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:40.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:40.438717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:50.668 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:50.656005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:50.681 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:50.668831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:50.731 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:01:50.719047 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:01:50.705564431 +0000 UTC m=+7316.524925824 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:02:00.903 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:00.890886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:00.915 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:00.903141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:11.134 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:11.121670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:11.146 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:11.134208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:21.374 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:21.360830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:21.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:21.371734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:31.606 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:31.593236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:31.620 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:31.607227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:41.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:41.837672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:41.859 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:41.847220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:52.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:52.068108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:52.094 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:02:52.082094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:02.322 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:02.309400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:02.370 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:02.357344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:12.598 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:12.583972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:12.634 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:12.621160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:22.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:22.837018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:22.859 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:22.846350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:33.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:33.064471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:33.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:33.076374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:43.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:43.299448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:43.326 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:43.313092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:53.549 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:53.535765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:53.561 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:03:53.548011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:03.787 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:03.773438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:03.798 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:03.784975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:14.015 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:14.001411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:14.029 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:14.015860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:24.271 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:24.258322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:24.282 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:24.268955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:34.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:34.490997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:34.516 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:34.502401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:44.736 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:44.722607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:44.744 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:44.731457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:54.984 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:54.970557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:54.992 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:04:54.979404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:05.223 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:05.209403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:05.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:05.256082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:15.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:15.478030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:15.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:15.490732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:25.725 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:25.711382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:25.736 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:25.722981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:35.958 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:35.944361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:35.972 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:35.958986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:46.191 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:46.177557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:46.204 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:46.191188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:56.426 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:56.413211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:56.439 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:05:56.425948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:06.659 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:06.645920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:06.673 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:06.659658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:16.894 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:16.881008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:16.907 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:16.893363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:27.129 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:27.115599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:27.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:27.127287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:37.412 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:37.398334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:37.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:37.411028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:47.642 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:47.629409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:47.655 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:47.642071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:57.874 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:57.861483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:57.887 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:06:57.874204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:08.127 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:08.114243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:08.141 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:08.128246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:18.359 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:18.346600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:18.373 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:18.360755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:28.599 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:28.585921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:28.610 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:28.597867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:38.827 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:38.814732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:38.839 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:38.826671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:49.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:49.089757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:49.116 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:49.102627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:59.372 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:59.359311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:59.383 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:07:59.370948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:09.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:09.601868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:09.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:09.612149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:19.842 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:19.829134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:19.854 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:19.842264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:30.073 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:30.060452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:30.085 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:30.072164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:40.324 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:40.311534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:40.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:40.331953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:50.591 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:50.579149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:50.602 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:08:50.590100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:00.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:00.808898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:00.834 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:00.821608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:11.053 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:11.040698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:11.065 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:11.052605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:21.283 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:21.270475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:21.297 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:21.283999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:31.518 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:31.506102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:31.547 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:31.529241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:41.768 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:41.756272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:41.778 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:41.765519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:47.628 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:47.616111 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T01:09:48.258 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:48.245260 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:09:48.236+0000 7f922fb81340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:09:48.258 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:09:48.236+0000 7f922fb81340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:09:48.259 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:09:48.236+0000 7f922fb81340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:09:48.259 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:09:48.259 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:48.245321 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:09:48.236+0000 7f922fb81340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:09:48.259 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:09:48.236+0000 7f922fb81340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:09:48.260 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:09:48.236+0000 7f922fb81340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:09:48.260 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:09:52.014 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:52.002023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:52.026 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:09:52.014859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:02.251 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:02.238987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:02.262 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:02.250642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:12.485 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:12.473469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:12.560 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:12.547935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:23.003 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:22.990892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:23.137 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:22.999371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:33.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:33.375756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:33.399 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:33.387107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:43.690 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:43.677701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:43.705 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:43.693450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:53.926 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:53.914595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:53.939 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:10:53.927091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:04.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:04.162701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:04.183 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:04.171532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:14.440 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:14.428457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:14.452 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:14.440683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:24.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:24.666860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:24.687 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:24.675450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:34.932 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:34.920316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:34.945 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:34.933636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:45.165 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:45.153098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:45.176 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:45.163943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:55.408 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:55.395525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:55.420 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:11:55.408908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:05.640 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:05.628170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:05.652 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:05.641069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:15.879 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:15.866728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:15.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:15.879468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:26.111 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:26.099472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:26.126 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:26.112571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:36.387 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:36.374901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:36.442 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:36.430492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:46.709 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:46.697913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:46.723 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:46.711573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:56.942 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:56.930673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:56.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:12:56.942794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:07.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:07.162188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:07.187 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:07.174737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:17.406 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:17.394493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:17.420 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:17.408370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:27.640 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:27.628718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:27.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:27.638902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:37.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:37.860095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:37.884 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:37.872339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:48.106 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:48.094248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:48.117 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:48.106150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:58.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:58.327755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:58.349 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:13:58.337909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:08.569 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:08.557495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:08.581 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:08.570032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:18.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:18.793128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:18.818 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:18.806660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:29.043 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:29.030234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:29.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:29.044964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:39.273 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:39.261754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:39.285 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:39.274170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:49.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:49.523982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:49.557 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:49.546539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:59.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:59.760751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:59.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:14:59.791968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:10.022 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:10.010792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:10.034 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:10.023119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:20.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:20.252821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:20.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:20.302310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:30.590 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:30.578951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:30.600 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:30.588957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:40.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:40.808171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:40.833 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:40.822249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:51.053 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:51.041679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:51.065 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:15:51.055095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:01.295 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:01.283861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:01.305 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:01.294118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:11.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:11.515726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:11.540 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:11.529216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:21.787 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:21.775188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:21.799 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:21.787140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:32.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:32.131671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:32.159 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:32.148278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:42.414 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:42.403022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:42.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:42.416505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:52.709 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:52.698374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:52.721 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:16:52.710780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:02.948 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:02.937479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:02.959 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:02.948253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:13.250 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:13.239204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:13.263 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:13.252972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:23.480 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:23.469585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:23.494 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:23.483530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:33.800 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:33.788842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:33.813 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:33.802746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:44.033 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:44.022233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:44.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:44.034059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:54.261 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:54.251069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:54.272 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:17:54.262473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:04.487 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:04.476482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:04.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:04.486679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:04.706 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:04.696148 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:18:04.685216686 +0000 UTC m=+8290.504578112 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:18:14.726 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:14.715535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:14.738 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:14.727327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:24.999 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:24.988529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:25.030 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:25.019960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:35.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:35.235645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:35.334 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:35.323888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:45.603 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:45.592930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:45.611 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:45.601642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:55.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:55.862393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:55.887 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:18:55.876789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:06.102 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:06.091730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:06.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:06.104319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:16.336 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:16.325708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:16.348 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:16.337866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:26.566 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:26.555640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:26.580 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:26.570366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:36.803 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:36.792195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:36.814 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:36.803890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:47.036 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:47.026062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:47.055 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:47.043104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:57.268 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:57.257729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:57.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:19:57.269945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:07.501 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:07.490892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:07.513 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:07.503109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:17.733 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:17.722966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:17.744 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:17.734309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:27.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:27.953078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:27.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:27.963974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:38.192 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:38.181937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:38.204 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:38.194303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:48.424 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:48.413893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:48.438 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:48.428219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:58.655 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:58.645312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:58.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:20:58.669605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:08.930 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:08.919467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:08.942 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:08.932783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:19.162 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:19.151342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:19.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:19.164586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:29.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:29.384660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:29.407 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:29.397102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:39.633 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:39.622137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:39.647 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:39.636751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:49.867 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:49.857013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:49.878 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:21:49.868991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:00.093 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:00.083376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:00.105 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:00.095341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:10.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:10.335637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:10.356 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:10.346263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:20.577 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:20.566967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:20.593 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:20.583667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:30.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:30.809146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:30.861 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:30.851569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:41.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:41.066842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:41.085 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:41.075907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:51.305 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:51.294801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:51.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:22:51.307879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:01.538 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:01.526930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:01.554 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:01.544338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:11.769 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:11.759274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:11.780 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:11.770570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:22.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:22.032313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:22.054 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:22.044965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:32.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:32.295490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:32.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:32.304044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:42.534 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:42.524589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:42.545 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:42.535934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:52.758 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:52.748862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:52.774 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:23:52.764416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:02.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:02.984842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:03.009 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:02.999348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:13.231 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:13.221233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:13.243 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:13.234089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:23.479 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:23.469927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:23.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:23.481981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:33.712 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:33.702335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:33.725 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:33.715692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:43.945 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:43.935146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:43.958 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:43.948755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:54.178 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:54.168807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:54.190 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:24:54.180765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:04.416 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:04.406974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:04.429 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:04.419496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:14.649 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:14.639342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:14.663 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:14.653333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:24.884 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:24.874805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:24.897 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:24.887830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:35.116 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:35.106714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:35.128 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:35.119092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:45.375 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:45.365628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:45.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:45.375117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:55.600 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:55.590559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:55.609 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:25:55.599648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:05.831 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:05.821368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:05.844 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:05.834684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:16.058 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:16.048400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:16.071 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:16.062198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:26.294 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:26.284111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:26.309 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:26.299749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:28.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:28.254460 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T01:26:28.871 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:28.861588 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:26:28.850+0000 7f2b540e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:26:28.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:26:28.850+0000 7f2b540e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:26:28.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:26:28.850+0000 7f2b540e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:26:28.872 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:26:28.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:28.861609 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:26:28.850+0000 7f2b540e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:26:28.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:26:28.850+0000 7f2b540e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:26:28.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:26:28.850+0000 7f2b540e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:26:28.873 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:26:36.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:36.517660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:36.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:36.551533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:46.793 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:46.783632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:46.808 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:46.798831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:57.058 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:57.048436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:57.071 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:26:57.061649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:07.285 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:07.275836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:07.296 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:07.286981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:17.512 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:17.502317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:17.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:17.575968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:27.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:27.794607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:27.814 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:27.805466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:38.034 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:38.025089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:38.048 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:38.038535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:48.271 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:48.261175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:48.283 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:48.273620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:58.506 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:58.496716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:58.534 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:27:58.525274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:08.759 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:08.749850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:08.836 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:08.826442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:19.053 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:19.043588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:19.158 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:19.148896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:29.377 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:29.367456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:29.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:29.379071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:39.604 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:39.594052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:39.617 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:39.607364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:49.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:49.862418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:49.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:28:49.873476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:00.104 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:00.095074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:00.114 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:00.105305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:10.335 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:10.324666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:10.346 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:10.336466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:20.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:20.554409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:20.574 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:20.564894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:30.795 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:30.785718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:30.805 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:30.795499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:41.026 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:41.016794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:41.042 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:41.029368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:51.260 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:51.250309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:51.273 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:29:51.263605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:01.546 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:01.536223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:01.560 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:01.550537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:11.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:11.783010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:11.803 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:11.793886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:22.025 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:22.015138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:22.038 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:22.029460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:32.268 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:32.258994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:32.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:32.272114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:42.500 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:42.490566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:42.513 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:42.503816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:52.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:52.763023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:52.797 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:30:52.788299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:03.017 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:03.007708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:03.030 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:03.020741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:13.251 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:13.241144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:13.262 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:13.253024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:23.486 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:23.476753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:23.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:23.485824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:33.718 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:33.708606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:33.731 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:33.721539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:43.956 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:43.946086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:43.967 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:43.958233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:54.186 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:54.176660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:54.198 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:31:54.187659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:04.418 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:04.408749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:04.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:04.422118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:14.663 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:14.653723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:14.675 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:14.665650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:24.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:24.881756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:24.903 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:24.894272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:35.136 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:35.127037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:35.151 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:35.141768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:45.370 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:45.360191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:45.383 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:45.373404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:55.603 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:55.592882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:55.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:32:55.604812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:05.831 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:05.821718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:05.839 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:05.829985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:16.062 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:16.051860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:16.073 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:16.063801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:26.298 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:26.288845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:26.312 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:26.301762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:36.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:36.525941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:36.548 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:36.538873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:46.763 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:46.753328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:46.779 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:46.769382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:57.003 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:56.993134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:57.015 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:33:57.004878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:07.236 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:07.225967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:07.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:07.238007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:17.467 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:17.457100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:17.481 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:17.471547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:27.701 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:27.691200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:27.714 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:27.704403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:37.933 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:37.922985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:37.993 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:37.983643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:37.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:37.983685 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:48.216 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:48.206713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:48.227 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:48.218267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:58.499 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:58.489565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:58.508 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:34:58.498437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:08.729 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:08.718978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:08.744 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:08.734030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:18.967 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:18.957008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:18.980 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:18.970816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:29.200 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:29.190088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:29.209 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:29.199783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:39.427 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:39.417968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:39.439 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:39.429239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:49.713 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:49.703267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:49.721 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:49.711781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:59.940 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:59.930010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:59.952 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:35:59.942311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:10.174 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:10.163796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:10.186 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:10.176385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:20.451 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:20.441010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:20.463 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:20.453769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:20.791 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:20.781417 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:36:20.74080274 +0000 UTC m=+9386.560164163 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:36:30.694 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:30.684253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:30.705 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:30.695647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:40.951 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:40.941212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:40.963 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:40.953128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:51.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:51.170469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:51.194 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:36:51.183929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:01.445 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:01.435449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:01.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:01.449181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:11.678 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:11.667996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:11.688 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:11.678217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:21.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:21.742849 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:37:21.707019042 +0000 UTC m=+9447.526380408 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:37:21.907 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:21.897471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:21.918 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:21.908975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:32.140 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:32.129695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:32.152 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:32.142101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:42.387 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:42.377316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:42.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:42.389969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:52.627 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:52.616481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:52.637 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:37:52.626975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:02.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:02.849601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:02.874 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:02.863752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:13.101 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:13.091571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:13.112 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:13.102216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:23.346 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:23.335576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:23.357 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:23.347193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:33.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:33.640014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:33.661 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:33.651325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:43.898 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:43.888049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:43.911 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:43.901326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:54.203 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:54.193226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:54.215 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:38:54.205004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:04.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:04.430625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:04.453 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:04.443239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:14.668 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:14.657855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:14.684 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:14.673685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:24.904 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:24.894181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:24.920 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:24.909850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:35.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:35.131503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:35.153 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:35.142741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:45.379 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:45.368566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:45.389 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:45.379382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:55.620 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:55.609713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:55.642 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:39:55.625536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:05.871 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:05.860644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:05.880 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:05.869706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:16.103 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:16.092270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:16.114 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:16.103948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:26.367 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:26.356393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:26.377 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:26.366625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:36.596 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:36.585643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:36.608 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:36.597833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:46.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:46.826627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:46.847 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:46.837199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:57.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:57.070407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:57.088 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:40:57.077646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:07.332 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:07.322172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:07.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:07.334594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:17.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:17.559255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:17.587 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:17.576817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:27.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:27.793130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:27.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:27.806908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:38.036 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:38.025174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:38.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:38.039181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:48.268 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:48.257135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:48.280 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:48.270150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:58.508 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:58.497733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:58.521 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:41:58.510849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:08.741 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:08.729837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:08.753 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:08.743328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:18.974 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:18.963032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:18.985 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:18.975074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:29.211 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:29.200810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:29.219 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:29.209067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:39.439 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:39.426873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:39.449 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:39.438570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:49.669 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:49.658923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:49.681 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:49.670979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:59.897 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:59.885749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:59.910 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:42:59.899589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:08.900 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:08.888892 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T01:43:09.496 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:09.485144 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:43:09.476+0000 7fae9d983340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:43:09.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:43:09.476+0000 7fae9d983340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:43:09.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:43:09.476+0000 7fae9d983340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:43:09.497 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:43:09.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:09.485171 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:43:09.476+0000 7fae9d983340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:43:09.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:43:09.476+0000 7fae9d983340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:43:09.498 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:43:09.476+0000 7fae9d983340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:43:09.498 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:43:10.128 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:10.117515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:10.140 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:10.129004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:20.359 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:20.347874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:20.371 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:20.360284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:30.587 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:30.575859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:30.598 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:30.587482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:40.815 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:40.804442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:40.883 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:40.872371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:51.100 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:51.089333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:51.113 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:43:51.103068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:01.329 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:01.318457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:01.341 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:01.330538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:11.560 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:11.549251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:11.575 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:11.563738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:21.800 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:21.786330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:21.809 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:21.798011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:32.025 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:32.013927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:32.037 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:32.026385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:42.270 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:42.259145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:42.284 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:42.273101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:52.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:52.503379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:52.526 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:44:52.515250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:02.747 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:02.735779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:02.759 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:02.748079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:12.979 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:12.967929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:12.991 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:12.980269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:23.226 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:23.214996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:23.238 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:23.227788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:33.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:33.447559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:33.471 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:33.460173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:43.712 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:43.700077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:43.721 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:43.709624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:53.951 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:53.939804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:53.962 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:45:53.951649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:04.194 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:04.183082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:04.206 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:04.195484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:14.425 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:14.413728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:14.436 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:14.425418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:24.658 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:24.646559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:24.670 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:24.658484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:34.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:34.913619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:34.940 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:34.929124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:45.157 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:45.145737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:45.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:45.159851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:55.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:55.383390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:55.407 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:46:55.395729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:05.648 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:05.636303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:05.666 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:05.653883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:15.880 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:15.868775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:15.895 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:15.883849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:26.111 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:26.099010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:26.123 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:26.111747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:36.342 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:36.330538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:36.359 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:36.347942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:46.617 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:46.605399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:46.630 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:46.618934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:56.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:56.837958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:56.859 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:47:56.848057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:07.079 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:07.067493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:07.090 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:07.078153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:17.318 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:17.306592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:17.335 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:17.323196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:27.554 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:27.542853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:27.564 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:27.552734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:37.797 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:37.785165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:37.813 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:37.801666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:48.059 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:48.047002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:48.067 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:48.055714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:58.284 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:58.272128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:58.294 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:48:58.282487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:08.516 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:08.504116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:08.530 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:08.518393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:18.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:18.734013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:18.758 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:18.746064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:28.978 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:28.966330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:28.991 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:28.978749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:39.212 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:39.199518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:39.225 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:39.213169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:49.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:49.470622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:49.496 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:49.483877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:59.716 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:59.704115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:59.729 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:49:59.717077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:09.975 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:09.962819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:09.993 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:09.980897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:20.217 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:20.205113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:20.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:20.218226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:30.451 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:30.438868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:30.465 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:30.452893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:40.687 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:40.674969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:40.697 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:40.685430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:50.920 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:50.907738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:50.976 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:50:50.963665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:01.223 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:01.210700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:01.234 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:01.222396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:11.455 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:11.442513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:11.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:11.472244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:21.704 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:21.691652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:21.716 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:21.703915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:31.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:31.936972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:31.959 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:31.946781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:42.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:42.167850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:42.190 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:42.178503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:52.414 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:52.401194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:52.425 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:51:52.412928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:02.647 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:02.634438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:02.662 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:02.650023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:12.892 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:12.879390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:12.903 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:12.890431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:23.124 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:23.111263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:23.135 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:23.122993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:33.357 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:33.344571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:33.368 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:33.356275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:43.600 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:43.587065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:43.613 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:43.600385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:53.864 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:53.851356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:53.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:52:53.859700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:04.092 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:04.079502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:04.103 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:04.091018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:14.323 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:14.310796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:14.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:14.326945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:24.560 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:24.545043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:24.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:24.556683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:34.792 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:34.779704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:34.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:34.791873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:45.027 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:45.014683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:45.047 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:45.032755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:55.270 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:55.257449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:55.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:53:55.268598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:05.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:05.501358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:05.527 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:05.514219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:15.755 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:15.742490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:15.770 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:15.757306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:25.992 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:25.979749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:26.004 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:25.991575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:36.225 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:36.211822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:36.285 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:36.273214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:46.505 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:46.492914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:46.519 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:46.507467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:56.752 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:56.740163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:56.764 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:54:56.751616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:06.983 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:06.971238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:06.997 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:06.984602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:17.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:17.242209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:17.265 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:17.253106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:27.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:27.491888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:27.515 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:27.502254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:37.758 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:37.745967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:37.769 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:37.756715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:37.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:37.956138 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:55:37.945751977 +0000 UTC m=+10543.765113432 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:55:47.985 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:47.972093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:47.997 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:47.984542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:58.254 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:58.241978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:58.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:55:58.254352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:08.487 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:08.474927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:08.500 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:08.488077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:18.722 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:18.709815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:18.735 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:18.721625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:28.957 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:28.945322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:28.967 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:28.955252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:39.183 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:39.171378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:39.193 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:39.180887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:49.425 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:49.412275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:49.481 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:49.468750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:59.698 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:59.685606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:59.707 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:56:59.695027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:09.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:09.912903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:09.940 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:09.927536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:20.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:20.148704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:20.172 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:20.160046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:30.393 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:30.380408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:30.478 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:30.392763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:40.629 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:40.616934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:40.637 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:40.625123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:50.856 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:50.844027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:50.868 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:57:50.856526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:01.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:01.077081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:01.101 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:01.089850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:11.322 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:11.310108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:11.335 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:11.323189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:21.559 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:21.547074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:21.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:21.558312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:31.789 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:31.776758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:31.821 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:31.809480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:42.060 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:42.048095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:42.073 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:42.061354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:52.293 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:52.281363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:52.319 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:58:52.307070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:02.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:02.522814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:02.546 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:02.533962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:12.812 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:12.799598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:12.824 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:12.812473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:23.060 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:23.048114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:23.070 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:23.058423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:33.289 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:33.276875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:33.299 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:33.287756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:43.523 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:43.510565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:43.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:43.523698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:49.504 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:49.492047 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T01:59:50.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:50.233461 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:59:50.223+0000 7f6761c73340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:59:50.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:59:50.223+0000 7f6761c73340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:59:50.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:59:50.223+0000 7f6761c73340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:59:50.247 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:59:50.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:50.233490 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:59:50.223+0000 7f6761c73340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:59:50.247 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:59:50.223+0000 7f6761c73340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:59:50.248 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T01:59:50.223+0000 7f6761c73340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:59:50.248 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:59:53.756 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:53.744474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:53.770 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 01:59:53.757734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:03.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:03.975259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:04.000 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:03.988202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:14.221 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:14.209328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:14.234 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:14.222018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:24.450 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:24.438217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:24.460 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:24.449065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:34.684 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:34.672072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:34.696 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:34.684797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:44.916 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:44.903858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:44.928 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:44.916696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:55.151 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:55.138929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:55.163 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:00:55.151315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:05.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:05.371652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:05.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:05.384663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:15.623 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:15.610648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:15.637 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:15.625456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:25.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:25.870895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:25.895 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:25.883678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:36.160 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:36.148632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:36.175 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:36.161830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:46.397 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:46.385631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:46.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:46.399941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:56.632 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:56.620575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:56.645 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:01:56.633254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:06.862 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:06.850098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:06.878 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:06.866094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:17.098 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:17.086276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:17.113 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:17.101190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:27.332 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:27.320207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:27.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:27.332674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:37.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:37.558339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:37.583 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:37.571443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:47.806 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:47.794777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:47.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:47.806008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:58.038 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:58.026745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:58.051 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:02:58.039242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:08.267 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:08.255544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:08.308 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:08.268467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:18.497 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:18.485614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:18.508 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:18.497002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:28.737 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:28.725147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:28.750 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:28.735617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:39.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:38.998504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:39.023 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:39.012320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:49.274 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:49.262197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:49.282 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:49.270931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:59.502 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:59.490236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:59.513 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:03:59.501256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:09.733 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:09.721601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:09.747 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:09.735456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:19.972 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:19.960528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:19.983 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:19.972531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:30.206 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:30.195001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:30.220 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:30.207669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:40.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:40.429153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:40.483 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:40.472306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:50.710 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:50.698621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:50.724 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:04:50.712053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:00.982 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:00.971153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:00.996 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:00.985066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:11.298 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:11.286945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:11.311 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:11.299415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:21.551 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:21.539772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:21.562 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:21.551441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:31.790 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:31.777377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:31.798 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:31.787579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:42.017 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:42.005462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:42.030 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:42.018857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:52.246 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:52.234552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:52.256 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:05:52.244705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:02.474 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:02.463170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:02.488 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:02.476978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:12.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:12.709032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:12.732 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:12.720803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:22.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:22.937972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:22.961 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:22.949942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:33.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:33.169141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:33.193 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:33.182167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:43.466 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:43.455262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:43.482 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:43.470659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:53.715 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:53.704256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:53.731 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:06:53.720400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:03.972 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:03.960506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:03.983 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:03.971897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:14.302 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:14.290723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:14.315 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:14.303472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:24.550 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:24.538713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:24.562 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:24.551131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:34.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:34.793200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:34.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:34.805956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:45.060 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:45.049768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:45.071 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:45.059570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:55.345 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:55.333918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:55.358 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:07:55.347113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:05.583 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:05.571464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:05.593 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:05.582418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:15.921 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:15.910635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:15.936 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:15.924620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:26.220 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:26.209128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:26.232 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:26.221001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:36.448 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:36.437553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:36.461 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:36.450881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:46.688 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:46.676376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:46.700 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:46.689410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:56.918 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:56.906843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:56.929 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:08:56.918768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:07.149 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:07.138312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:07.160 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:07.149982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:17.556 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:17.545389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:17.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:17.559064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:27.875 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:27.864298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:27.890 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:27.879211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:38.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:38.098506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:38.122 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:38.111255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:48.340 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:48.328788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:48.351 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:48.340692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:58.569 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:58.557927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:58.578 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:09:58.568012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:08.796 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:08.785644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:08.807 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:08.796934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:19.088 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:19.077772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:19.104 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:19.092145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:29.321 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:29.310763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:29.331 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:29.320933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:39.630 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:39.619331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:39.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:39.627938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:49.889 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:49.878813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:49.901 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:10:49.890301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:00.123 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:00.109958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:00.131 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:00.120679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:10.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:10.367015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:10.390 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:10.380119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:20.616 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:20.603687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:20.800 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:20.789600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:31.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:31.075633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:31.099 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:31.088507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:41.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:41.400181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:41.441 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:41.430923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:51.888 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:51.877371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:51.900 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:51.889623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:52.162 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:11:52.151391 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:11:52.14115161 +0000 UTC m=+11517.960513071 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:12:02.130 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:02.119068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:02.142 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:02.131043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:12.402 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:12.389620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:12.409 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:12.398657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:22.630 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:22.619674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:22.643 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:22.632417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:32.863 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:32.852267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:32.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:32.866455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:43.104 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:43.093267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:43.116 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:43.105889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:53.402 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:53.391716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:53.414 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:12:53.404282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:03.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:03.628571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:03.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:03.640220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:13.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:13.861273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:13.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:13.876208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:24.104 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:24.093705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:24.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:24.104846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:34.350 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:34.339239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:34.363 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:34.353030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:44.587 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:44.577009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:44.597 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:44.587076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:54.873 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:54.862274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:54.883 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:13:54.873232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:05.104 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:05.093266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:05.115 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:05.104558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:15.338 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:15.327836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:15.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:15.389608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:25.621 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:25.610317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:25.636 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:25.626298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:35.880 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:35.869632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:35.889 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:35.878478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:46.111 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:46.100271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:46.123 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:46.112867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:56.343 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:56.332392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:56.356 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:14:56.346159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:06.574 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:06.563200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:06.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:06.575405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:16.807 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:16.796728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:16.820 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:16.809656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:27.048 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:27.038209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:27.089 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:27.077910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:37.347 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:37.337161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:37.360 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:37.349569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:47.586 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:47.575585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:47.601 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:47.590667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:57.819 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:57.809034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:57.830 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:15:57.820453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:08.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:08.038803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:08.062 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:08.051636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:18.316 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:18.305968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:18.328 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:18.318378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:28.548 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:28.538003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:28.561 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:28.551082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:30.271 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:30.260765 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T02:16:30.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:30.975664 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:16:30.968+0000 7fe674c44340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:16:30.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:16:30.968+0000 7fe674c44340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:16:30.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:16:30.968+0000 7fe674c44340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:16:30.988 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:16:30.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:30.975688 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:16:30.968+0000 7fe674c44340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:16:30.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:16:30.968+0000 7fe674c44340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:16:30.988 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:16:30.968+0000 7fe674c44340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:16:30.989 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:16:38.781 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:38.771393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:38.789 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:38.779866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:49.015 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:49.004286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:49.028 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:49.017802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:59.250 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:59.239403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:59.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:16:59.253585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:09.484 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:09.474418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:09.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:09.485091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:19.718 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:19.707713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:19.730 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:19.720436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:29.954 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:29.943727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:29.967 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:29.957206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:40.191 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:40.180416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:40.205 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:40.195435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:50.429 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:50.418938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:50.440 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:17:50.430088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:00.720 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:00.709503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:00.761 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:00.751532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:10.995 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:10.984739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:11.009 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:10.998790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:21.261 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:21.251469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:21.272 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:21.262314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:31.491 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:31.481044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:31.505 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:31.495353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:41.727 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:41.716854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:41.740 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:41.729699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:51.986 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:51.975946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:51.999 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:18:51.988938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:02.218 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:02.207637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:02.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:02.220043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:12.454 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:12.443723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:12.466 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:12.456357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:22.690 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:22.679445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:22.703 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:22.693568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:32.929 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:32.918923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:32.940 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:32.930259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:43.172 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:43.161605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:43.180 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:43.170456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:53.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:53.389913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:53.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:19:53.401612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:03.688 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:03.677626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:03.699 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:03.689203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:13.958 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:13.948190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:13.972 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:13.962008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:24.190 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:24.179575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:24.202 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:24.191819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:34.422 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:34.412240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:34.433 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:34.423409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:44.659 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:44.648949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:44.671 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:44.660988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:54.891 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:54.880799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:54.932 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:20:54.921804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:05.159 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:05.143492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:05.165 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:05.155277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:15.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:15.378300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:15.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:15.390502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:25.623 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:25.612614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:25.631 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:25.621213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:35.850 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:35.840195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:35.859 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:35.849609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:46.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:46.070796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:46.093 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:46.083623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:56.361 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:56.350538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:56.380 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:21:56.370225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:06.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:06.615250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:06.636 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:06.626648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:16.856 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:16.846081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:16.868 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:16.858529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:27.158 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:27.147622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:27.166 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:27.156248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:37.386 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:37.376297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:37.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:37.386900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:47.615 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:47.605589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:47.630 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:47.620279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:57.855 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:57.844432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:57.866 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:22:57.856369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:08.080 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:08.070469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:08.093 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:08.083934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:18.351 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:18.341126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:18.365 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:18.355507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:28.590 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:28.580007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:28.601 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:28.591631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:38.829 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:38.819003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:38.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:38.827627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:49.068 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:49.058528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:49.082 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:49.072415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:59.300 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:59.290606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:59.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:23:59.303383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:09.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:09.525427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:09.545 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:09.535364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:19.853 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:19.843288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:19.868 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:19.858339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:30.128 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:30.118141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:30.167 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:30.157831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:40.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:40.377695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:40.404 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:40.394755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:50.624 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:50.614280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:50.638 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:24:50.627413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:00.862 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:00.852102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:00.874 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:00.863749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:11.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:11.086653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:11.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:11.098561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:21.365 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:21.354816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:21.378 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:21.368443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:31.601 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:31.591015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:31.611 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:31.601907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:41.841 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:41.830758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:41.851 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:41.841453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:52.082 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:52.072206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:52.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:25:52.087606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:02.330 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:02.320366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:02.393 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:02.382987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:12.653 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:12.643458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:12.666 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:12.656226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:22.892 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:22.881785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:22.910 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:22.900046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:33.131 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:33.120503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:33.143 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:33.132097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:43.360 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:43.349633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:43.371 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:43.361753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:53.595 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:53.584880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:53.610 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:26:53.600212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:03.833 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:03.822579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:03.844 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:03.834310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:14.066 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:14.056014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:14.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:14.071664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:24.297 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:24.287077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:24.307 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:24.297593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:34.528 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:34.517552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:34.542 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:34.532269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:44.759 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:44.748569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:44.771 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:44.760896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:54.990 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:54.978459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:54.999 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:27:54.988933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:05.219 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:05.208767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:05.230 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:05.220597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:15.450 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:15.439843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:15.462 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:15.451871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:25.679 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:25.669127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:25.692 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:25.682392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:35.909 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:35.898438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:35.919 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:35.909880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:46.144 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:46.134112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:46.153 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:46.143621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:56.384 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:56.373759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:56.396 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:28:56.385881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:06.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:06.604427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:06.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:06.615182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:16.843 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:16.832674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:16.854 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:16.844597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:27.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:27.067274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:27.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:27.076557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:37.324 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:37.314264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:37.336 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:37.326769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:47.558 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:47.548074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:47.570 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:47.559978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:57.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:57.826640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:57.847 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:29:57.837560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:08.010 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:07.999973 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:30:07.993839647 +0000 UTC m=+12613.813201014 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:30:08.143 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:08.130749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:08.154 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:08.144819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:18.380 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:18.369915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:18.389 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:18.379502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:28.654 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:28.643950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:28.667 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:28.657126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:38.890 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:38.880347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:38.903 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:38.892681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:49.125 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:49.114598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:49.137 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:49.125732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:59.379 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:59.369030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:59.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:30:59.381692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:09.643 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:09.632801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:09.654 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:09.643833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:19.869 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:19.859256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:19.881 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:19.871542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:30.109 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:30.097809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:30.165 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:30.154334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:40.395 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:40.384827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:40.411 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:40.401062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:50.634 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:50.624090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:50.649 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:31:50.639002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:00.924 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:00.914090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:00.938 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:00.927632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:11.157 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:11.146586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:11.169 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:11.159076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:21.385 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:21.374323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:21.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:21.390065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:31.615 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:31.604761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:31.628 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:31.617628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:41.844 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:41.833456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:41.857 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:41.847089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:52.074 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:52.063370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:52.087 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:32:52.077355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:02.317 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:02.306944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:02.329 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:02.318604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:10.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:10.983223 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T02:33:11.613 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:11.602013 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:33:11.591+0000 7f8d32b80340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:33:11.613 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:33:11.591+0000 7f8d32b80340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:33:11.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:33:11.591+0000 7f8d32b80340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:33:11.614 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:33:11.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:11.602047 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:33:11.591+0000 7f8d32b80340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:33:11.614 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:33:11.591+0000 7f8d32b80340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:33:11.615 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:33:11.591+0000 7f8d32b80340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:33:11.615 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:33:12.574 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:12.564029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:12.586 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:12.575664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:22.804 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:22.793801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:22.814 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:22.804329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:33.045 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:33.034234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:33.056 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:33.045949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:43.278 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:43.267110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:43.289 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:43.279401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:53.512 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:53.501014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:53.524 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:33:53.514022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:03.771 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:03.760128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:03.789 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:03.778545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:14.041 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:14.030421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:14.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:14.039485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:24.273 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:24.262276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:24.284 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:24.273842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:34.507 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:34.495967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:34.520 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:34.509768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:44.781 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:44.770392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:44.789 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:44.778646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:44.790 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:44.778673 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:55.011 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:55.000515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:55.022 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:34:55.010863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:05.238 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:05.227934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:05.250 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:05.240048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:15.467 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:15.456238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:15.478 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:15.467716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:25.698 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:25.687596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:25.706 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:25.696106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:35.923 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:35.913031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:35.935 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:35.925074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:46.156 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:46.145572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:46.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:46.160504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:56.390 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:56.379378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:56.401 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:35:56.390641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:06.622 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:06.611366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:06.636 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:06.625628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:16.874 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:16.863678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:16.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:16.876142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:27.108 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:27.097764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:27.120 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:27.109181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:37.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:37.328029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:37.351 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:37.340978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:47.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:47.635817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:47.660 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:47.649726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:57.882 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:57.871019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:57.892 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:36:57.882058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:08.112 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:08.101718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:08.128 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:08.117963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:18.355 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:18.344528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:18.369 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:18.358502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:28.602 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:28.591594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:28.616 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:28.605462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:38.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:38.826386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:38.847 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:38.837468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:49.070 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:49.059539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:49.081 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:49.071124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:59.351 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:59.340752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:59.365 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:37:59.354990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:09.632 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:09.622078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:09.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:09.635691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:19.866 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:19.855076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:19.879 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:19.868489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:30.101 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:30.090532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:30.112 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:30.102233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:40.334 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:40.323255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:40.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:40.334371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:50.563 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:50.552432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:50.575 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:38:50.564661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:00.796 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:00.786108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:00.808 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:00.798075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:11.030 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:11.019316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:11.044 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:11.033924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:21.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:21.258896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:21.313 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:21.299591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:31.535 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:31.524644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:31.548 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:31.538063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:41.765 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:41.755343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:41.775 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:41.764162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:52.025 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:52.015495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:52.037 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:39:52.027004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:02.316 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:02.305516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:02.329 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:02.318928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:12.575 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:12.564540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:12.588 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:12.577958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:22.817 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:22.806876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:22.828 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:22.818477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:33.049 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:33.038940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:33.062 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:33.051813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:43.284 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:43.274548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:43.300 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:43.289881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:53.571 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:53.561547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:53.651 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:40:53.641182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:03.872 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:03.862034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:03.886 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:03.875440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:14.110 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:14.100286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:14.120 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:14.110775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:24.389 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:24.378621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:24.412 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:24.402117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:34.631 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:34.620521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:34.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:34.635139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:44.934 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:44.924051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:44.946 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:44.935443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:55.202 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:55.191649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:55.212 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:41:55.202612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:05.431 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:05.421099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:05.440 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:05.430869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:15.690 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:15.680084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:15.703 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:15.693237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:25.929 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:25.918984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:25.942 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:25.932545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:36.159 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:36.149176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:36.172 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:36.162345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:46.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:46.382582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:46.407 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:46.395570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:56.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:56.615409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:56.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:42:56.629811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:06.858 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:06.848316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:06.867 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:06.857967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:17.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:17.076808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:17.097 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:17.087238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:27.328 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:27.318784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:27.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:27.329270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:37.572 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:37.562028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:37.585 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:37.575395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:47.830 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:47.820974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:47.845 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:47.835667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:58.078 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:58.068877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:58.092 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:43:58.082447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:08.310 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:08.300434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:08.323 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:08.313748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:18.541 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:18.531794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:18.553 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:18.543419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:28.772 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:28.762690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:28.785 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:28.775411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:39.001 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:38.991571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:39.013 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:39.003939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:49.232 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:49.222587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:49.242 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:49.232900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:59.460 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:59.450954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:59.470 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:44:59.461441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:09.693 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:09.683942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:09.704 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:09.695345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:19.925 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:19.915103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:19.937 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:19.927907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:30.157 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:30.147902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:30.170 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:30.161225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:40.390 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:40.376890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:40.397 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:40.388013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:50.635 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:50.626108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:50.648 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:45:50.639510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:00.876 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:00.866446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:00.887 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:00.878419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:11.132 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:11.122576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:11.143 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:11.134045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:21.375 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:21.365993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:21.388 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:21.378580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:31.605 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:31.595576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:31.616 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:31.607135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:41.835 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:41.825908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:41.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:41.839345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:52.068 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:52.058667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:52.077 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:46:52.068012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:02.298 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:02.288182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:02.311 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:02.302167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:12.533 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:12.524051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:12.549 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:12.540084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:22.767 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:22.757728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:22.818 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:22.808790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:23.073 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:23.063973 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:47:23.057363474 +0000 UTC m=+13648.876724921 LastTransitionTime:2022-03-08 23:01:52 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:47:33.040 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:33.029737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:33.055 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:33.045822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:43.290 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:43.281584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:43.302 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:43.293399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:53.534 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:53.524504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:53.546 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:47:53.537327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:03.762 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:03.753347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:03.777 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:03.768555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:13.996 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:13.987045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:14.005 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:13.996710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:24.238 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:24.228750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:24.249 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:24.240654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:34.468 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:34.458863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:34.481 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:34.472598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:44.700 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:44.690912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:44.713 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:44.704345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:54.934 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:54.925129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:54.944 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:48:54.935345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:05.162 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:05.152912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:05.172 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:05.163838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:15.392 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:15.383295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:15.405 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:15.396425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:25.637 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:25.628443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:25.646 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:25.637372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:35.866 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:35.857129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:35.877 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:35.867809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:46.094 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:46.085758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:46.105 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:46.097011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:51.658 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:51.649388 I | op-mon: parsing mon endpoints: a=10.106.34.234:6789 2022-03-09T02:49:52.265 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:52.254442 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:49:52.245+0000 7f29db2b7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:49:52.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:49:52.245+0000 7f29db2b7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:49:52.267 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:49:52.245+0000 7f29db2b7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:49:52.267 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:49:52.268 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:52.254468 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:49:52.245+0000 7f29db2b7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:49:52.268 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:49:52.245+0000 7f29db2b7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:49:52.269 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09T02:49:52.245+0000 7f29db2b7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:49:52.269 INFO:tasks.rook.operator.smithi117.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:49:56.330 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:56.321448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:56.344 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:49:56.335268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:06.586 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:06.577953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:06.599 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:06.589452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:16.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:16.851299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:16.869 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:16.860424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:27.088 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:27.079250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:27.133 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:27.124598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:37.360 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:37.351687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:37.419 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:37.409773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:47.639 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:47.630384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:47.650 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:47.641724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:57.938 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:57.929600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:57.949 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:50:57.941241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:08.171 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:08.161811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:08.181 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:08.172964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:18.403 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:18.394350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:18.417 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:18.407366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:28.637 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:28.628470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:28.649 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:28.641310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:38.871 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:38.862905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:38.884 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:38.875644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:49.127 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:49.118845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:49.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:49.151562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:59.437 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:59.428336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:59.448 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:51:59.439798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:09.672 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:09.662957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:09.685 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:09.677270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:19.909 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:19.899878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:19.927 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:19.917890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:30.150 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:30.142023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:30.164 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:30.155390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:40.385 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:40.376686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:40.400 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:40.392003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:50.618 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:50.609503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:50.632 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:52:50.623826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:00.852 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:00.843842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:00.864 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:00.856243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:11.086 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:11.078121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:11.098 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:11.089914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:21.325 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:21.316996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:21.339 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:21.329506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:31.565 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:31.557257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:31.578 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:31.570133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:41.802 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:41.794383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:41.813 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:41.805445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:52.037 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:52.028669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:52.048 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:53:52.040596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:02.264 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:02.256259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:02.276 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:02.268183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:12.499 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:12.490795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:12.510 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:12.502098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:22.728 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:22.720228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:22.737 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:22.728709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:32.955 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:32.946025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:32.966 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:32.957725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:43.186 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:43.177881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:43.199 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:43.191149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:53.421 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:53.412757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:53.433 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:54:53.425145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:03.683 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:03.675330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:03.697 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:03.688752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:13.915 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:13.906894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:13.927 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:13.919118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:24.149 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:24.141019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:24.161 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:24.153666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:34.385 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:34.377124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:34.403 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:34.393344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:44.625 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:44.616722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:44.636 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:44.628886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:54.851 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:54.843650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:55:54.860 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:55:54.852644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:05.079 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:05.071086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:05.088 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:05.080135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:15.372 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:15.364369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:15.385 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:15.377232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:25.608 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:25.599771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:25.619 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:25.611312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:35.837 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:35.828724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:35.848 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:35.840794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:46.071 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:46.063284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:46.083 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:46.075750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:56.304 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:56.295784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:56:56.321 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:56:56.313844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:06.544 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:06.536535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:06.553 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:06.545602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:16.771 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:16.763761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:16.785 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:16.778001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:27.008 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:27.000257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:27.020 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:27.012736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:37.281 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:37.273715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:37.294 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:37.285840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:47.514 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:47.506487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:47.526 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:47.519085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:57.748 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:57.740745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:57:57.760 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:57:57.753096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:07.994 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:07.986841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:08.008 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:08.000309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:18.266 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:18.258432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:18.277 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:18.269459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:28.495 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:28.487273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:28.506 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:28.498701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:38.725 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:38.717599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:38.738 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:38.730743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:48.957 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:48.949236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:48.968 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:48.961210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:59.263 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:59.255033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:58:59.278 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:58:59.270142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:09.499 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:09.491640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:09.511 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:09.503629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:19.733 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:19.725409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:19.746 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:19.738947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:29.973 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:29.965581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:29.987 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:29.980009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:40.211 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:40.203793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:40.223 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:40.215957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:50.459 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:50.452042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:59:50.471 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 02:59:50.464166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T03:00:00.692 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 03:00:00.684608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T03:00:00.706 INFO:tasks.rook.operator.smithi117.stdout:2022-03-09 03:00:00.698939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T05:20:51.504 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:20:51.514 DEBUG:teuthology.task.console_log:Killing console logger for smithi117 2022-03-09T05:20:51.515 DEBUG:teuthology.exit:Finished running handlers