2022-03-08T00:23:30.629 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T00:23:30.631 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} --verbose --archive /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725493 -- /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725493/orig.config.yaml 2022-03-08T00:23:30.667 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T00:23:30.733 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725493 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} email: ceph-qa@ceph.io first_in_suite: false job_id: '6725493' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxX8fPllwnuSLOlotGSXM34K67HNa3kLz8xWN8q/i9dj/tEEqOZZRnHWnj8CX1fRkP+UyHLNy4PWgpdbgXTe/ePExSZIokO/JsUl00+Gwh/DMMGw0x2NLSTUOeeahhylFkSyBEDQBJKNENkaixZQCssQiBUbqqkP65Qc/stQyxfeoeN8eGfbuUE78NzVcwqW2j6sXKbyEnDx3FqhY4UbfozwTDe0qQ6x7YF05f+osWSQ7kodoQ8PGplSdNWToQEii1M6cUapRntoJE4SiTDPNG9RHnldUy03msIeQlzx9qSRjlX2VyHT7aBYigQbob4AKrzbH1LBqfKQ0uXevJAgpzlaCbrdjzjoYlVBHDfpKlcO+bctfpJ1tPIpPQqsQyG1ks4Mk4uh+JMxTHgrEaNHFseY8tAEvLVKzWhr9+MqoXANqOPl7JG6NRuS2Q8MqoL+U4fRUq6BAvIh7WoiNpGIzzfbPkxEfUBjWwmiQtOWYclUpM/8utQvKmAvUDRtTCegc= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-07_22:09:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T00:23:30.734 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-08T00:23:30.735 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-08T00:23:30.735 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T00:23:30.737 INFO:teuthology.task.internal:Checking packages... 2022-03-08T00:23:30.758 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-08T00:23:30.759 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T00:23:30.759 INFO:teuthology.packaging:ref: None 2022-03-08T00:23:30.760 INFO:teuthology.packaging:tag: None 2022-03-08T00:23:30.760 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-08T00:23:30.760 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T00:23:30.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-08T00:23:30.922 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T00:23:30.923 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T00:23:30.935 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T00:23:30.935 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T00:23:30.995 INFO:teuthology.task.internal:Saving configuration 2022-03-08T00:23:31.008 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T00:23:31.058 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T00:23:31.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725493', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 00:18:11.211487', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIPkOniIienbNT8M86E4tTR1LyqAa4YpEbPmPvJNtHp1'} 2022-03-08T00:23:31.082 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T00:23:31.188 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T00:23:31.188 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T00:23:31.252 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdd471ff9d8>, signals=[15]) 2022-03-08T00:23:31.253 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T00:23:31.262 INFO:teuthology.task.internal:Opening connections... 2022-03-08T00:23:31.262 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2022-03-08T00:23:31.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T00:23:31.312 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T00:23:31.349 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2022-03-08T00:23:31.433 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2022-03-08T00:23:31.433 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2022-03-08T00:23:31.480 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Ubuntu" 2022-03-08T00:23:31.481 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T00:23:31.481 INFO:teuthology.orchestra.run.smithi120.stdout:ID=ubuntu 2022-03-08T00:23:31.481 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE=debian 2022-03-08T00:23:31.481 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T00:23:31.482 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="20.04" 2022-03-08T00:23:31.482 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T00:23:31.483 INFO:teuthology.orchestra.run.smithi120.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T00:23:31.483 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T00:23:31.484 INFO:teuthology.orchestra.run.smithi120.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T00:23:31.484 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_CODENAME=focal 2022-03-08T00:23:31.484 INFO:teuthology.orchestra.run.smithi120.stdout:UBUNTU_CODENAME=focal 2022-03-08T00:23:31.485 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2022-03-08T00:23:31.505 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T00:23:31.522 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T00:23:31.538 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T00:23:31.539 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2022-03-08T00:23:31.545 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T00:23:31.554 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T00:23:31.554 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2022-03-08T00:23:31.572 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T00:23:31.603 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T00:23:31.618 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T00:23:31.619 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T00:23:31.619 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-08T00:23:31.619 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T00:23:31.620 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2022-03-08T00:23:31.627 INFO:teuthology.orchestra.run.smithi120.stdout:5.4.0-97-generic 2022-03-08T00:23:31.627 INFO:teuthology.task.kernel:Running kernel on smithi120: 5.4.0-97-generic 2022-03-08T00:23:31.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get clean 2022-03-08T00:23:31.705 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get update 2022-03-08T00:23:32.008 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T00:23:32.008 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T00:23:32.065 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T00:23:32.408 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T00:23:32.472 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-08T00:23:32.617 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-08T00:23:32.742 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-08T00:23:32.857 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-08T00:23:32.894 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-08T00:23:32.917 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T00:23:32.917 INFO:teuthology.orchestra.run.smithi120.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-08T00:23:32.962 INFO:teuthology.orchestra.run.smithi120.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T00:23:32.963 INFO:teuthology.orchestra.run.smithi120.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-08T00:23:32.967 INFO:teuthology.orchestra.run.smithi120.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T00:23:32.968 INFO:teuthology.orchestra.run.smithi120.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-08T00:23:32.992 INFO:teuthology.orchestra.run.smithi120.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T00:23:33.006 INFO:teuthology.orchestra.run.smithi120.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T00:23:33.010 INFO:teuthology.orchestra.run.smithi120.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T00:23:33.015 INFO:teuthology.orchestra.run.smithi120.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-08T00:23:33.028 INFO:teuthology.orchestra.run.smithi120.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T00:23:33.029 INFO:teuthology.orchestra.run.smithi120.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-08T00:23:33.062 INFO:teuthology.orchestra.run.smithi120.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T00:23:33.062 INFO:teuthology.orchestra.run.smithi120.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T00:23:33.063 INFO:teuthology.orchestra.run.smithi120.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T00:23:33.064 INFO:teuthology.orchestra.run.smithi120.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T00:23:33.066 INFO:teuthology.orchestra.run.smithi120.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T00:23:33.067 INFO:teuthology.orchestra.run.smithi120.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-08T00:23:33.071 INFO:teuthology.orchestra.run.smithi120.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T00:23:33.072 INFO:teuthology.orchestra.run.smithi120.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-08T00:23:33.104 INFO:teuthology.orchestra.run.smithi120.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-08T00:23:33.120 INFO:teuthology.orchestra.run.smithi120.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-08T00:23:33.123 INFO:teuthology.orchestra.run.smithi120.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T00:23:33.154 INFO:teuthology.orchestra.run.smithi120.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T00:23:33.155 INFO:teuthology.orchestra.run.smithi120.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T00:23:33.155 INFO:teuthology.orchestra.run.smithi120.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T00:23:33.156 INFO:teuthology.orchestra.run.smithi120.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T00:23:33.156 INFO:teuthology.orchestra.run.smithi120.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T00:23:33.180 INFO:teuthology.orchestra.run.smithi120.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T00:23:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T00:23:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T00:23:33.263 INFO:teuthology.orchestra.run.smithi120.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T00:23:33.263 INFO:teuthology.orchestra.run.smithi120.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T00:23:33.263 INFO:teuthology.orchestra.run.smithi120.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T00:23:35.975 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 10.0 MB in 2s (4,682 kB/s) 2022-03-08T00:23:37.161 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:23:37.182 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T00:23:37.248 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:23:37.420 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:23:37.420 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:23:37.560 INFO:teuthology.orchestra.run.smithi120.stdout:The following additional packages will be installed: 2022-03-08T00:23:37.561 INFO:teuthology.orchestra.run.smithi120.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T00:23:37.561 INFO:teuthology.orchestra.run.smithi120.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-08T00:23:37.561 INFO:teuthology.orchestra.run.smithi120.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-08T00:23:37.562 INFO:teuthology.orchestra.run.smithi120.stdout:Suggested packages: 2022-03-08T00:23:37.562 INFO:teuthology.orchestra.run.smithi120.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T00:23:37.562 INFO:teuthology.orchestra.run.smithi120.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-08T00:23:37.611 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-03-08T00:23:37.611 INFO:teuthology.orchestra.run.smithi120.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T00:23:37.611 INFO:teuthology.orchestra.run.smithi120.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-08T00:23:37.612 INFO:teuthology.orchestra.run.smithi120.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-08T00:23:37.612 INFO:teuthology.orchestra.run.smithi120.stdout: thermald upower usbmuxd 2022-03-08T00:23:38.367 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T00:23:38.368 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 80.1 MB of archives. 2022-03-08T00:23:38.368 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T00:23:38.368 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T00:23:38.637 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T00:23:38.652 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T00:23:38.653 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T00:23:38.740 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T00:23:38.741 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-08T00:23:39.655 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-08T00:23:39.918 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-08T00:23:41.558 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-08T00:23:41.558 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T00:23:41.559 INFO:teuthology.orchestra.run.smithi120.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T00:23:41.564 INFO:teuthology.orchestra.run.smithi120.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T00:23:41.567 INFO:teuthology.orchestra.run.smithi120.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T00:23:41.954 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 80.1 MB in 4s (20.3 MB/s) 2022-03-08T00:23:42.128 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T00:23:43.610 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T00:23:43.615 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T00:23:43.749 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T00:23:44.085 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T00:23:44.099 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T00:23:44.144 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T00:23:44.463 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T00:23:44.478 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T00:23:44.521 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T00:23:44.874 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T00:23:44.890 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T00:23:44.933 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T00:23:45.311 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T00:23:45.324 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T00:23:45.360 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T00:23:45.647 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-08T00:23:45.660 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T00:23:45.688 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:23:47.938 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-08T00:23:47.952 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T00:23:48.060 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:23:48.549 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-08T00:23:48.562 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T00:23:48.591 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:23:54.656 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T00:23:54.670 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-08T00:23:54.706 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T00:23:55.030 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T00:23:55.043 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T00:23:55.088 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T00:23:55.450 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package thermald. 2022-03-08T00:23:55.463 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T00:23:55.508 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T00:23:56.075 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package upower. 2022-03-08T00:23:56.088 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T00:23:56.143 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T00:23:56.593 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T00:23:56.607 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T00:23:56.651 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T00:23:56.933 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T00:23:57.081 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T00:23:57.207 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T00:23:57.333 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T00:23:57.451 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T00:23:57.552 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T00:23:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T00:23:58.145 INFO:teuthology.orchestra.run.smithi120.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T00:23:58.197 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T00:23:58.367 INFO:teuthology.orchestra.run.smithi120.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T00:23:58.381 INFO:teuthology.orchestra.run.smithi120.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T00:23:58.381 INFO:teuthology.orchestra.run.smithi120.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T00:23:58.847 INFO:teuthology.orchestra.run.smithi120.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T00:23:59.082 INFO:teuthology.orchestra.run.smithi120.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T00:23:59.119 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T00:23:59.428 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T00:23:59.429 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T00:23:59.924 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:24:01.178 INFO:teuthology.orchestra.run.smithi120.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-08T00:24:01.178 INFO:teuthology.orchestra.run.smithi120.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-08T00:24:01.327 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:24:02.662 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:24:03.963 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T00:24:04.089 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T00:24:09.095 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T00:24:09.756 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T00:24:09.875 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T00:24:10.080 INFO:teuthology.orchestra.run.smithi120.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T00:24:10.080 INFO:teuthology.orchestra.run.smithi120.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-08T00:24:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T00:24:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:Sourcing file `/etc/default/grub' 2022-03-08T00:24:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T00:24:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:Generating grub configuration file ... 2022-03-08T00:24:33.814 INFO:teuthology.orchestra.run.smithi120.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T00:24:33.847 INFO:teuthology.orchestra.run.smithi120.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T00:24:34.221 INFO:teuthology.orchestra.run.smithi120.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T00:24:34.233 INFO:teuthology.orchestra.run.smithi120.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T00:24:36.696 INFO:teuthology.orchestra.run.smithi120.stdout:done 2022-03-08T00:24:37.041 DEBUG:teuthology.orchestra.run.smithi120:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T00:24:37.071 INFO:teuthology.orchestra.run.smithi120.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T00:24:37.072 INFO:teuthology.orchestra.run.smithi120.stdout:Status: install ok installed 2022-03-08T00:24:37.072 INFO:teuthology.orchestra.run.smithi120.stdout:Priority: optional 2022-03-08T00:24:37.072 INFO:teuthology.orchestra.run.smithi120.stdout:Section: kernel 2022-03-08T00:24:37.073 INFO:teuthology.orchestra.run.smithi120.stdout:Installed-Size: 19 2022-03-08T00:24:37.073 INFO:teuthology.orchestra.run.smithi120.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T00:24:37.074 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: amd64 2022-03-08T00:24:37.074 INFO:teuthology.orchestra.run.smithi120.stdout:Source: linux-meta-hwe-5.13 2022-03-08T00:24:37.075 INFO:teuthology.orchestra.run.smithi120.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T00:24:37.075 INFO:teuthology.orchestra.run.smithi120.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T00:24:37.076 INFO:teuthology.orchestra.run.smithi120.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T00:24:37.076 INFO:teuthology.orchestra.run.smithi120.stdout:Recommends: thermald 2022-03-08T00:24:37.076 INFO:teuthology.orchestra.run.smithi120.stdout:Description: Generic Linux kernel image 2022-03-08T00:24:37.077 INFO:teuthology.orchestra.run.smithi120.stdout: This package will always depend on the latest generic kernel image 2022-03-08T00:24:37.077 INFO:teuthology.orchestra.run.smithi120.stdout: available. 2022-03-08T00:24:37.078 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-08T00:24:37.078 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T00:24:37.079 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T00:24:37.079 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T00:24:37.080 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T00:24:37.080 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get clean 2022-03-08T00:24:37.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get update 2022-03-08T00:24:37.362 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T00:24:37.453 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T00:24:37.535 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T00:24:37.617 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T00:24:38.884 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:24:38.908 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T00:24:38.978 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:24:39.150 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:24:39.151 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:24:39.325 INFO:teuthology.orchestra.run.smithi120.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-08T00:24:39.325 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T00:24:39.327 DEBUG:teuthology.orchestra.run.smithi120:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T00:24:39.346 INFO:teuthology.orchestra.run.smithi120.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T00:24:39.347 INFO:teuthology.orchestra.run.smithi120.stdout:Status: install ok installed 2022-03-08T00:24:39.347 INFO:teuthology.orchestra.run.smithi120.stdout:Priority: optional 2022-03-08T00:24:39.347 INFO:teuthology.orchestra.run.smithi120.stdout:Section: kernel 2022-03-08T00:24:39.348 INFO:teuthology.orchestra.run.smithi120.stdout:Installed-Size: 19 2022-03-08T00:24:39.348 INFO:teuthology.orchestra.run.smithi120.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T00:24:39.348 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: amd64 2022-03-08T00:24:39.349 INFO:teuthology.orchestra.run.smithi120.stdout:Source: linux-meta-hwe-5.13 2022-03-08T00:24:39.349 INFO:teuthology.orchestra.run.smithi120.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T00:24:39.349 INFO:teuthology.orchestra.run.smithi120.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T00:24:39.349 INFO:teuthology.orchestra.run.smithi120.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T00:24:39.350 INFO:teuthology.orchestra.run.smithi120.stdout:Recommends: thermald 2022-03-08T00:24:39.350 INFO:teuthology.orchestra.run.smithi120.stdout:Description: Generic Linux kernel image 2022-03-08T00:24:39.350 INFO:teuthology.orchestra.run.smithi120.stdout: This package will always depend on the latest generic kernel image 2022-03-08T00:24:39.350 INFO:teuthology.orchestra.run.smithi120.stdout: available. 2022-03-08T00:24:39.351 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2022-03-08T00:24:39.357 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.IMhmcxMsry 2022-03-08T00:24:39.358 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IMhmcxMsry 2022-03-08T00:24:39.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.IMhmcxMsry 2022-03-08T00:24:39.452 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.IMhmcxMsry is 8KB 2022-03-08T00:24:39.469 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.IMhmcxMsry 2022-03-08T00:24:39.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T00:24:39.532 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:24:39.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T00:24:39.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T00:24:39.569 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-08T00:24:39.570 DEBUG:teuthology.orchestra.run.smithi120:> sudo update-grub 2022-03-08T00:24:40.845 INFO:teuthology.orchestra.run.smithi120.stderr:Sourcing file `/etc/default/grub' 2022-03-08T00:24:40.851 INFO:teuthology.orchestra.run.smithi120.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T00:24:40.873 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2022-03-08T00:24:41.158 INFO:teuthology.orchestra.run.smithi120.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T00:24:41.170 INFO:teuthology.orchestra.run.smithi120.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T00:24:41.528 INFO:teuthology.orchestra.run.smithi120.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T00:24:41.540 INFO:teuthology.orchestra.run.smithi120.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T00:24:42.919 INFO:teuthology.orchestra.run.smithi120.stderr:done 2022-03-08T00:24:42.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2022-03-08T00:25:12.951 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T00:25:12.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-03-08T00:25:12.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T00:25:31.339 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2022-03-08T00:26:01.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-03-08T00:26:01.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T00:26:04.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2022-03-08T00:26:34.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-03-08T00:26:34.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T00:26:34.956 DEBUG:teuthology.orchestra.run.smithi120:> true 2022-03-08T00:26:35.426 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-03-08T00:26:35.426 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-08T00:26:35.427 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2022-03-08T00:26:35.432 INFO:teuthology.orchestra.run.smithi120.stdout:5.13.0-30-generic 2022-03-08T00:26:35.433 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-08T00:26:35.433 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T00:26:35.433 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T00:26:36.435 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T00:26:36.435 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T00:26:36.496 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2022-03-08T00:26:36.524 DEBUG:teuthology.parallel:result is None 2022-03-08T00:26:36.525 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T00:26:36.536 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T00:26:36.537 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T00:26:36.544 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T00:26:36.557 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T00:26:36.572 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T00:26:36.572 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T00:26:36.596 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T00:26:36.607 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T00:26:36.608 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T00:26:36.657 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T00:26:36.667 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T00:26:36.668 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T00:26:36.679 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T00:26:36.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T00:26:36.704 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T00:26:36.724 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T00:26:36.725 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T00:26:36.752 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T00:26:36.759 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T00:26:36.767 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:26:36.767 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T00:26:36.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2022-03-08T00:26:36.876 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T00:26:36.883 INFO:teuthology.task.internal:Starting timer... 2022-03-08T00:26:36.884 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T00:26:37.140 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T00:26:37.235 DEBUG:teuthology.task.selinux:Excluding smithi120: OS 'ubuntu' does not support SELinux 2022-03-08T00:26:37.236 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T00:26:37.236 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T00:26:37.237 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T00:26:37.237 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T00:26:37.341 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T00:26:37.342 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T00:26:37.362 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T00:26:37.365 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T00:29:36.584 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2022-03-08T00:29:36.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-03-08T00:29:36.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T00:29:36.642 DEBUG:teuthology.orchestra.run.smithi120:> true 2022-03-08T00:29:36.760 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T00:29:36.771 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T00:29:36.772 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T00:29:36.774 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T00:29:36.807 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T00:29:36.808 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Command line: ntpd -gq 2022-03-08T00:29:36.808 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: proto: precision = 0.092 usec (-23) 2022-03-08T00:29:36.809 INFO:teuthology.orchestra.run.smithi120.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T00:29:36.809 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T00:29:36.809 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T00:29:36.810 INFO:teuthology.orchestra.run.smithi120.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T00:29:36.811 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T00:29:36.811 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T00:29:36.812 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T00:29:36.812 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen normally on 3 enp3s0f1 172.21.15.120:123 2022-03-08T00:29:36.812 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen normally on 4 lo [::1]:123 2022-03-08T00:29:36.812 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:478f%5]:123 2022-03-08T00:29:36.812 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:36 ntpd[8999]: Listening on routing socket on fd #22 for interface updates 2022-03-08T00:29:37.808 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:37.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:37.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:37.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:37.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:37.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:37 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:38.808 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:38.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:38.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:38.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:38.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:38.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:38 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:39.808 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:39.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:39.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:39.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:39.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:39.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:39 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:40.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:40.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:40.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:40.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:40.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:40.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:40.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:40.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:40.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:40.814 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:40.814 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:40.815 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:40 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:41.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:41.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:41.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:41.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:41.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:41.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:41.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:41.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:41.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:41.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:41.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:41.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:41 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:42.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:42.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:42.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:42.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:42.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:42.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:42.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:42.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:42.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:42.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:42.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:42.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:42 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:43.809 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:43.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:43.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:43.810 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:43.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:43.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:43.811 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:43.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:43.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T00:29:43.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T00:29:43.812 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T00:29:43.813 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T00:29:43.813 INFO:teuthology.orchestra.run.smithi120.stdout: 8 Mar 00:29:43 ntpd[8999]: ntpd: time set +0.245847 s 2022-03-08T00:29:43.813 INFO:teuthology.orchestra.run.smithi120.stdout:ntpd: time set +0.245847s 2022-03-08T00:29:43.814 INFO:teuthology.orchestra.run.smithi120.stderr: 8 Mar 00:29:43 ntpd[8999]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T00:29:43.867 INFO:teuthology.orchestra.run.smithi120.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T00:29:43.868 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================== 2022-03-08T00:29:43.868 INFO:teuthology.orchestra.run.smithi120.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T00:29:43.868 INFO:teuthology.orchestra.run.smithi120.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T00:29:43.868 INFO:teuthology.orchestra.run.smithi120.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T00:29:43.869 INFO:teuthology.orchestra.run.smithi120.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T00:29:43.870 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T00:29:43.886 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T00:29:43.887 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-08T00:29:43.887 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T00:29:43.941 INFO:teuthology.orchestra.run.smithi120.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T00:29:43.950 INFO:teuthology.orchestra.run.smithi120.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T00:29:43.958 INFO:teuthology.orchestra.run.smithi120.stdout:net.ipv4.ip_forward = 1 2022-03-08T00:29:44.012 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:29:44.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T00:29:44.027 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:29:44.027 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/docker/daemon.json 2022-03-08T00:29:44.041 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl restart docker || true 2022-03-08T00:29:45.087 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T00:29:45.093 INFO:tasks.kubeadm:172.21.15.120 is in 172.21.0.0/20 at pos 3959 2022-03-08T00:29:45.107 INFO:tasks.kubeadm:Pod subnet: 10.251.184.0/21 2022-03-08T00:29:45.108 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T00:29:45.108 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T00:29:45.138 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:45.138 INFO:teuthology.orchestra.run.smithi120.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T00:29:45.138 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:45.382 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T00:29:45.383 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T00:29:45.453 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T00:29:45.535 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T00:29:46.874 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:29:47.048 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:29:47.049 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:29:47.139 INFO:teuthology.orchestra.run.smithi120.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T00:29:47.155 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:47.155 INFO:teuthology.orchestra.run.smithi120.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T00:29:47.155 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:47.211 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:29:47.385 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:29:47.385 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:29:47.558 INFO:teuthology.orchestra.run.smithi120.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T00:29:47.559 INFO:teuthology.orchestra.run.smithi120.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T00:29:47.559 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-03-08T00:29:47.560 INFO:teuthology.orchestra.run.smithi120.stdout: apt-transport-https 2022-03-08T00:29:47.747 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T00:29:47.748 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 4,680 B of archives. 2022-03-08T00:29:47.748 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T00:29:47.748 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T00:29:47.955 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T00:29:47.956 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T00:29:47.956 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: falling back to frontend: Readline 2022-03-08T00:29:47.976 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T00:29:47.976 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T00:29:47.976 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: falling back to frontend: Teletype 2022-03-08T00:29:47.982 INFO:teuthology.orchestra.run.smithi120.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T00:29:48.031 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 4,680 B in 0s (25.8 kB/s) 2022-03-08T00:29:48.120 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T00:29:48.162 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T00:29:48.164 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T00:29:48.210 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T00:29:48.483 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T00:29:49.014 INFO:teuthology.orchestra.run.smithi120.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T00:29:49.028 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:49.029 INFO:teuthology.orchestra.run.smithi120.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T00:29:49.029 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:49.167 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T00:29:49.281 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T00:29:49.363 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T00:29:49.446 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T00:29:49.837 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T00:29:49.930 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T00:29:50.068 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 63.3 kB in 1s (71.9 kB/s) 2022-03-08T00:29:50.905 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:29:51.073 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:29:51.074 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:29:51.165 INFO:teuthology.orchestra.run.smithi120.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T00:29:51.180 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:51.181 INFO:teuthology.orchestra.run.smithi120.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T00:29:51.181 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:29:51.234 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:29:51.411 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:29:51.411 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:29:51.552 INFO:teuthology.orchestra.run.smithi120.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T00:29:51.553 INFO:teuthology.orchestra.run.smithi120.stdout:bridge-utils set to manually installed. 2022-03-08T00:29:51.553 INFO:teuthology.orchestra.run.smithi120.stdout:The following additional packages will be installed: 2022-03-08T00:29:51.554 INFO:teuthology.orchestra.run.smithi120.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T00:29:51.554 INFO:teuthology.orchestra.run.smithi120.stdout:Suggested packages: 2022-03-08T00:29:51.555 INFO:teuthology.orchestra.run.smithi120.stdout: nftables 2022-03-08T00:29:51.597 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-03-08T00:29:51.598 INFO:teuthology.orchestra.run.smithi120.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T00:29:51.789 INFO:teuthology.orchestra.run.smithi120.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T00:29:51.790 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 73.6 MB of archives. 2022-03-08T00:29:51.790 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T00:29:51.790 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T00:29:51.825 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T00:29:51.976 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T00:29:52.118 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T00:29:52.399 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T00:29:53.519 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T00:29:54.105 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T00:29:54.367 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T00:29:54.880 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T00:29:54.880 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T00:29:54.880 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: falling back to frontend: Readline 2022-03-08T00:29:54.888 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T00:29:54.888 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T00:29:54.889 INFO:teuthology.orchestra.run.smithi120.stderr:debconf: falling back to frontend: Teletype 2022-03-08T00:29:54.894 INFO:teuthology.orchestra.run.smithi120.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T00:29:54.943 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 73.6 MB in 3s (24.1 MB/s) 2022-03-08T00:29:55.035 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package conntrack. 2022-03-08T00:29:55.079 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T00:29:55.082 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T00:29:55.125 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T00:29:55.420 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package cri-tools. 2022-03-08T00:29:55.434 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T00:29:55.470 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T00:29:56.565 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ebtables. 2022-03-08T00:29:56.577 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T00:29:56.631 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T00:29:56.901 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T00:29:56.915 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T00:29:56.960 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T00:29:58.362 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package socat. 2022-03-08T00:29:58.376 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T00:29:58.412 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T00:29:58.807 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package kubelet. 2022-03-08T00:29:58.820 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T00:29:58.857 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T00:30:01.306 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package kubectl. 2022-03-08T00:30:01.319 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T00:30:01.348 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T00:30:02.592 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package kubeadm. 2022-03-08T00:30:02.600 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T00:30:02.642 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T00:30:03.805 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T00:30:03.932 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T00:30:04.058 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T00:30:04.922 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T00:30:05.051 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T00:30:05.176 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T00:30:05.303 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T00:30:05.502 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T00:30:06.021 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T00:30:06.548 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T00:30:07.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T00:30:21.108 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T00:30:24.660 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T00:30:26.890 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T00:30:30.315 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T00:30:31.788 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T00:30:36.839 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T00:30:39.014 INFO:teuthology.orchestra.run.smithi120.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T00:30:39.018 INFO:tasks.kubeadm:Token: abcdef.98gw8eex9i46i3af 2022-03-08T00:30:39.018 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): '172.21.15.120'} 2022-03-08T00:30:39.019 DEBUG:teuthology.orchestra.run.smithi120:> sudo kubeadm init --node-name smithi120 --token abcdef.98gw8eex9i46i3af --pod-network-cidr 10.251.184.0/21 2022-03-08T00:30:39.213 INFO:teuthology.orchestra.run.smithi120.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T00:30:39.214 INFO:teuthology.orchestra.run.smithi120.stdout:[preflight] Running pre-flight checks 2022-03-08T00:30:39.616 INFO:teuthology.orchestra.run.smithi120.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T00:30:39.616 INFO:teuthology.orchestra.run.smithi120.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T00:30:39.616 INFO:teuthology.orchestra.run.smithi120.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T00:30:39.922 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T00:30:40.080 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "ca" certificate and key 2022-03-08T00:30:40.239 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T00:30:40.240 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi120] and IPs [10.96.0.1 172.21.15.120] 2022-03-08T00:30:40.445 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T00:30:40.557 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T00:30:40.785 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T00:30:40.873 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T00:30:41.064 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T00:30:41.065 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi120] and IPs [172.21.15.120 127.0.0.1 ::1] 2022-03-08T00:30:41.436 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T00:30:41.436 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi120] and IPs [172.21.15.120 127.0.0.1 ::1] 2022-03-08T00:30:41.625 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T00:30:41.817 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T00:30:42.099 INFO:teuthology.orchestra.run.smithi120.stdout:[certs] Generating "sa" key and public key 2022-03-08T00:30:42.100 INFO:teuthology.orchestra.run.smithi120.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T00:30:42.178 INFO:teuthology.orchestra.run.smithi120.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T00:30:42.225 INFO:teuthology.orchestra.run.smithi120.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T00:30:42.392 INFO:teuthology.orchestra.run.smithi120.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T00:30:42.500 INFO:teuthology.orchestra.run.smithi120.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T00:30:42.513 INFO:teuthology.orchestra.run.smithi120.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T00:30:42.515 INFO:teuthology.orchestra.run.smithi120.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T00:30:42.515 INFO:teuthology.orchestra.run.smithi120.stdout:[kubelet-start] Starting the kubelet 2022-03-08T00:30:42.743 INFO:teuthology.orchestra.run.smithi120.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T00:30:42.744 INFO:teuthology.orchestra.run.smithi120.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T00:30:42.747 INFO:teuthology.orchestra.run.smithi120.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T00:30:42.748 INFO:teuthology.orchestra.run.smithi120.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T00:30:42.749 INFO:teuthology.orchestra.run.smithi120.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T00:30:42.750 INFO:teuthology.orchestra.run.smithi120.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T00:30:59.753 INFO:teuthology.orchestra.run.smithi120.stdout:[apiclient] All control plane components are healthy after 17.003037 seconds 2022-03-08T00:30:59.754 INFO:teuthology.orchestra.run.smithi120.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T00:30:59.774 INFO:teuthology.orchestra.run.smithi120.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T00:30:59.775 INFO:teuthology.orchestra.run.smithi120.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T00:31:00.386 INFO:teuthology.orchestra.run.smithi120.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T00:31:00.386 INFO:teuthology.orchestra.run.smithi120.stdout:[mark-control-plane] Marking the node smithi120 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T00:31:00.386 INFO:teuthology.orchestra.run.smithi120.stdout:[mark-control-plane] Marking the node smithi120 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T00:31:00.895 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] Using token: abcdef.98gw8eex9i46i3af 2022-03-08T00:31:00.895 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T00:31:00.926 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T00:31:00.941 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T00:31:00.996 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T00:31:00.996 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T00:31:01.007 INFO:teuthology.orchestra.run.smithi120.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T00:31:01.049 INFO:teuthology.orchestra.run.smithi120.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T00:31:01.491 INFO:teuthology.orchestra.run.smithi120.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T00:31:01.574 INFO:teuthology.orchestra.run.smithi120.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T00:31:01.575 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.575 INFO:teuthology.orchestra.run.smithi120.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T00:31:01.575 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.576 INFO:teuthology.orchestra.run.smithi120.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T00:31:01.576 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.576 INFO:teuthology.orchestra.run.smithi120.stdout: mkdir -p $HOME/.kube 2022-03-08T00:31:01.576 INFO:teuthology.orchestra.run.smithi120.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T00:31:01.577 INFO:teuthology.orchestra.run.smithi120.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T00:31:01.577 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.577 INFO:teuthology.orchestra.run.smithi120.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T00:31:01.577 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.578 INFO:teuthology.orchestra.run.smithi120.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T00:31:01.578 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.578 INFO:teuthology.orchestra.run.smithi120.stdout:You should now deploy a pod network to the cluster. 2022-03-08T00:31:01.578 INFO:teuthology.orchestra.run.smithi120.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T00:31:01.578 INFO:teuthology.orchestra.run.smithi120.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T00:31:01.579 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.579 INFO:teuthology.orchestra.run.smithi120.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T00:31:01.579 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:31:01.579 INFO:teuthology.orchestra.run.smithi120.stdout:kubeadm join 172.21.15.120:6443 --token abcdef.98gw8eex9i46i3af \ 2022-03-08T00:31:01.580 INFO:teuthology.orchestra.run.smithi120.stdout: --discovery-token-ca-cert-hash sha256:0f5c6e9c99cc0864dec36ec1a7b1120cb93fb74e7afbd81cc78a8521acda7e8a 2022-03-08T00:31:01.581 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:01.581 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T00:31:01.592 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T00:31:01.592 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T00:31:01.603 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:01.604 DEBUG:teuthology.orchestra.run.smithi120:> dd of=.kube/config 2022-03-08T00:31:01.650 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:01.651 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/root/.kube/config 2022-03-08T00:31:01.667 DEBUG:teuthology.orchestra.run.smithi120:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T00:31:05.369 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T00:31:05.398 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T00:31:05.431 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T00:31:05.511 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T00:31:05.531 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T00:31:05.571 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T00:31:05.607 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T00:31:05.677 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T00:31:05.724 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T00:31:05.757 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T00:31:05.858 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T00:31:05.883 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T00:31:05.917 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T00:31:05.996 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T00:31:06.033 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T00:31:06.079 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T00:31:06.170 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T00:31:06.217 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T00:31:06.335 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T00:31:06.393 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T00:31:06.435 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T00:31:06.521 INFO:teuthology.orchestra.run.smithi120.stdout:namespace/tigera-operator created 2022-03-08T00:31:06.532 INFO:teuthology.orchestra.run.smithi120.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T00:31:06.560 INFO:teuthology.orchestra.run.smithi120.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T00:31:06.588 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/tigera-operator created 2022-03-08T00:31:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T00:31:06.686 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T00:31:06.703 INFO:teuthology.orchestra.run.smithi120.stdout:deployment.apps/tigera-operator created 2022-03-08T00:31:06.707 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f - 2022-03-08T00:31:09.491 INFO:teuthology.orchestra.run.smithi120.stdout:installation.operator.tigera.io/default created 2022-03-08T00:31:09.496 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T00:31:09.497 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:09.497 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T00:31:09.503 INFO:tasks.kubeadm:Scratch devices on smithi120: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T00:31:09.504 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T00:31:09.573 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records in 2022-03-08T00:31:09.573 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2022-03-08T00:31:09.574 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0167713 s, 625 MB/s 2022-03-08T00:31:09.575 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T00:31:09.600 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records in 2022-03-08T00:31:09.601 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2022-03-08T00:31:09.601 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138497 s, 757 MB/s 2022-03-08T00:31:09.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T00:31:09.628 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records in 2022-03-08T00:31:09.628 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2022-03-08T00:31:09.628 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015752 s, 666 MB/s 2022-03-08T00:31:09.629 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T00:31:09.655 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records in 2022-03-08T00:31:09.655 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2022-03-08T00:31:09.655 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140226 s, 748 MB/s 2022-03-08T00:31:09.663 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T00:31:09.664 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi120-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi120 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi120-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi120 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi120-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi120 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi120-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi120 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T00:31:09.664 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f - 2022-03-08T00:31:10.568 INFO:teuthology.orchestra.run.smithi120.stdout:persistentvolume/smithi120-lv-1 created 2022-03-08T00:31:10.573 INFO:teuthology.orchestra.run.smithi120.stdout:persistentvolume/smithi120-lv-2 created 2022-03-08T00:31:10.582 INFO:teuthology.orchestra.run.smithi120.stdout:persistentvolume/smithi120-lv-3 created 2022-03-08T00:31:10.608 INFO:teuthology.orchestra.run.smithi120.stdout:persistentvolume/smithi120-lv-4 created 2022-03-08T00:31:10.615 INFO:teuthology.orchestra.run.smithi120.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T00:31:10.618 DEBUG:teuthology.orchestra.run.smithi120:> kubectl taint node smithi120 node-role.kubernetes.io/master- || true 2022-03-08T00:31:10.689 INFO:teuthology.orchestra.run.smithi120.stdout:node/smithi120 untainted 2022-03-08T00:31:10.692 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T00:31:10.693 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T00:31:10.702 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T00:31:10.703 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:10.703 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T00:31:10.710 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T00:31:10.711 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T00:31:10.718 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T00:31:10.719 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T00:31:10.719 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T00:31:10.719 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T00:31:10.719 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-03-08 00:31:09.583887190 +0000 2022-03-08T00:31:10.720 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-03-08 00:31:09.579887190 +0000 2022-03-08T00:31:10.720 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-03-08 00:31:09.579887190 +0000 2022-03-08T00:31:10.720 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-03-08T00:31:10.725 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-03-08T00:31:10.725 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-03-08T00:31:10.725 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000554361 s, 924 kB/s 2022-03-08T00:31:10.729 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T00:31:10.735 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T00:31:10.736 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T00:31:10.736 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-03-08T00:31:10.736 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T00:31:10.737 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-03-08 00:31:09.603887190 +0000 2022-03-08T00:31:10.737 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-03-08 00:31:09.603887190 +0000 2022-03-08T00:31:10.737 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-03-08 00:31:09.603887190 +0000 2022-03-08T00:31:10.737 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-03-08T00:31:10.743 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-03-08T00:31:10.743 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-03-08T00:31:10.744 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000555964 s, 921 kB/s 2022-03-08T00:31:10.748 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T00:31:10.797 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T00:31:10.797 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T00:31:10.799 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T00:31:10.799 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T00:31:10.799 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-03-08 00:31:09.631887191 +0000 2022-03-08T00:31:10.800 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-03-08 00:31:09.631887191 +0000 2022-03-08T00:31:10.800 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-03-08 00:31:09.631887191 +0000 2022-03-08T00:31:10.800 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-03-08T00:31:10.804 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-03-08T00:31:10.804 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-03-08T00:31:10.804 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000509744 s, 1.0 MB/s 2022-03-08T00:31:10.808 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T00:31:10.815 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T00:31:10.815 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T00:31:10.815 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-08T00:31:10.815 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T00:31:10.816 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2022-03-08 00:31:09.659887191 +0000 2022-03-08T00:31:10.816 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2022-03-08 00:31:09.659887191 +0000 2022-03-08T00:31:10.816 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2022-03-08 00:31:09.659887191 +0000 2022-03-08T00:31:10.817 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2022-03-08T00:31:10.822 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2022-03-08T00:31:10.822 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2022-03-08T00:31:10.823 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.0005154 s, 993 kB/s 2022-03-08T00:31:10.826 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T00:31:10.921 INFO:teuthology.orchestra.run.smithi120.stdout:loop 2022-03-08T00:31:10.923 INFO:tasks.nvme_loop:Connecting nvme_loop smithi120:/dev/vg_nvme/lv_1... 2022-03-08T00:31:10.923 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T00:31:10.941 INFO:teuthology.orchestra.run.smithi120.stdout:1 2022-03-08T00:31:10.966 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_11 2022-03-08T00:31:10.988 INFO:tasks.nvme_loop:Connecting nvme_loop smithi120:/dev/vg_nvme/lv_2... 2022-03-08T00:31:10.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T00:31:11.006 INFO:teuthology.orchestra.run.smithi120.stdout:1 2022-03-08T00:31:11.028 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_21 2022-03-08T00:31:11.052 INFO:tasks.nvme_loop:Connecting nvme_loop smithi120:/dev/vg_nvme/lv_3... 2022-03-08T00:31:11.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T00:31:11.118 INFO:teuthology.orchestra.run.smithi120.stdout:1 2022-03-08T00:31:11.142 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_31 2022-03-08T00:31:11.165 INFO:tasks.nvme_loop:Connecting nvme_loop smithi120:/dev/vg_nvme/lv_4... 2022-03-08T00:31:11.166 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T00:31:11.187 INFO:teuthology.orchestra.run.smithi120.stdout:1 2022-03-08T00:31:11.207 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_41 2022-03-08T00:31:11.231 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:11.232 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T00:31:11.238 DEBUG:teuthology.orchestra.run.smithi120:> sudo nvme list 2022-03-08T00:31:11.300 INFO:teuthology.orchestra.run.smithi120.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T00:31:11.301 INFO:teuthology.orchestra.run.smithi120.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T00:31:11.301 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/nvme0n1 PHFT620400XV400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101F0 2022-03-08T00:31:11.301 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/nvme1n1 97581fa61e444f7b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T00:31:11.302 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/nvme2n1 beb5771cbe6e6f8a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T00:31:11.302 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/nvme3n1 9038ecc742e6d0c4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T00:31:11.302 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/nvme4n1 0458278926619e2f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T00:31:11.302 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T00:31:11.303 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:11.303 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/scratch_devs 2022-03-08T00:31:11.318 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T00:31:11.405 INFO:tasks.rook:Rook start 2022-03-08T00:31:11.406 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-08T00:31:11.406 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T00:31:11.407 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-08T00:31:11.407 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-08T00:31:11.416 INFO:teuthology.orchestra.run.smithi120.stderr:Cloning into 'rook'... 2022-03-08T00:31:13.514 INFO:teuthology.orchestra.run.smithi120.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-08T00:31:13.515 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.515 INFO:teuthology.orchestra.run.smithi120.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-08T00:31:13.515 INFO:teuthology.orchestra.run.smithi120.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-08T00:31:13.515 INFO:teuthology.orchestra.run.smithi120.stderr:state without impacting any branches by switching back to a branch. 2022-03-08T00:31:13.516 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.516 INFO:teuthology.orchestra.run.smithi120.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-08T00:31:13.516 INFO:teuthology.orchestra.run.smithi120.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-08T00:31:13.516 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.517 INFO:teuthology.orchestra.run.smithi120.stderr: git switch -c 2022-03-08T00:31:13.517 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.517 INFO:teuthology.orchestra.run.smithi120.stderr:Or undo this operation with: 2022-03-08T00:31:13.517 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.518 INFO:teuthology.orchestra.run.smithi120.stderr: git switch - 2022-03-08T00:31:13.518 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.518 INFO:teuthology.orchestra.run.smithi120.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-08T00:31:13.518 INFO:teuthology.orchestra.run.smithi120.stderr: 2022-03-08T00:31:13.618 INFO:tasks.rook:/home/teuthworker 2022-03-08T00:31:13.619 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T00:31:13.671 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T00:31:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T00:31:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T00:31:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2022-03-08T00:31:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2022-03-08T00:31:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2022-03-08T00:31:13.673 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T00:31:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2022-03-08T00:31:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2022-03-08T00:31:13.674 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T00:31:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2022-03-08T00:31:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2022-03-08T00:31:13.674 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T00:31:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2022-03-08T00:31:13.675 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T00:31:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2022-03-08T00:31:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T00:31:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42908sec preferred_lft 42908sec 2022-03-08T00:31:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2022-03-08T00:31:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2022-03-08T00:31:13.676 INFO:teuthology.orchestra.run.smithi120.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T00:31:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 02:42:c2:3c:10:da brd ff:ff:ff:ff:ff:ff 2022-03-08T00:31:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T00:31:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2022-03-08T00:31:13.678 DEBUG:teuthology.orchestra.run.smithi120:> which systemctl 2022-03-08T00:31:13.683 INFO:teuthology.orchestra.run.smithi120.stdout:/usr/bin/systemctl 2022-03-08T00:31:13.683 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T00:31:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T00:31:13.730 DEBUG:teuthology.orchestra.run.smithi120:> true 2022-03-08T00:31:13.760 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-03-08T00:31:13.809 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-03-08T00:31:13.814 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T00:31:13.815 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T00:31:13.835 DEBUG:teuthology.orchestra.remote:smithi120:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T00:31:13.850 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:13.851 DEBUG:teuthology.orchestra.run.smithi120:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-08T00:31:13.864 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-08T00:31:13.916 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:31:13.916 DEBUG:teuthology.orchestra.run.smithi120:> dd of=operator.yaml 2022-03-08T00:31:13.924 INFO:tasks.rook:Deploying operator 2022-03-08T00:31:13.943 DEBUG:teuthology.orchestra.remote:smithi120:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T00:31:14.021 DEBUG:teuthology.orchestra.remote:smithi120:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T00:31:14.036 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-08T00:31:14.276 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T00:31:14.301 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T00:31:14.434 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T00:31:14.464 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T00:31:14.529 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T00:31:14.551 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T00:31:14.581 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T00:31:14.735 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T00:31:14.760 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T00:31:14.785 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T00:31:14.811 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T00:31:14.905 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T00:31:14.918 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T00:31:14.940 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T00:31:14.951 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-08T00:31:14.969 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-08T00:31:14.985 INFO:teuthology.orchestra.run.smithi120.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-08T00:31:15.059 INFO:teuthology.orchestra.run.smithi120.stdout:namespace/rook-ceph created 2022-03-08T00:31:15.064 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T00:31:15.085 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-08T00:31:15.111 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-08T00:31:15.126 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-08T00:31:15.143 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T00:31:15.203 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T00:31:15.208 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T00:31:15.219 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T00:31:15.236 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T00:31:15.248 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T00:31:15.256 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-system created 2022-03-08T00:31:15.264 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T00:31:15.273 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T00:31:15.294 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T00:31:15.349 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T00:31:15.356 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T00:31:15.364 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T00:31:15.386 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T00:31:15.402 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T00:31:15.419 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T00:31:15.436 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T00:31:15.448 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T00:31:15.512 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T00:31:15.522 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T00:31:15.531 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T00:31:15.541 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T00:31:15.549 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T00:31:15.570 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T00:31:15.581 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T00:31:15.592 INFO:teuthology.orchestra.run.smithi120.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T00:31:15.592 INFO:teuthology.orchestra.run.smithi120.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T00:31:15.598 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T00:31:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T00:31:15.674 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T00:31:15.677 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T00:31:15.689 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T00:31:15.702 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T00:31:15.706 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T00:31:15.714 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T00:31:15.723 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T00:31:15.763 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T00:31:15.951 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T00:31:15.955 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T00:31:16.013 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T00:31:16.040 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T00:31:16.107 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T00:31:16.122 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T00:31:16.149 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T00:31:16.174 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T00:31:16.203 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T00:31:16.287 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T00:31:16.299 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T00:31:16.326 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T00:31:16.349 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T00:31:16.375 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T00:31:16.458 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T00:31:16.462 INFO:teuthology.orchestra.run.smithi120.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T00:31:16.473 INFO:teuthology.orchestra.run.smithi120.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T00:31:16.500 INFO:teuthology.orchestra.run.smithi120.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T00:31:16.524 INFO:teuthology.orchestra.run.smithi120.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T00:31:16.775 INFO:teuthology.orchestra.run.smithi120.stdout:configmap/rook-ceph-operator-config created 2022-03-08T00:31:17.203 INFO:teuthology.orchestra.run.smithi120.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T00:31:17.206 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:31:17.289 INFO:teuthology.orchestra.run.smithi120.stderr:No resources found in rook-ceph namespace. 2022-03-08T00:31:27.295 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:31:27.356 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:31:27.357 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 Pending 0 10s 2022-03-08T00:31:37.359 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:31:37.423 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:31:37.423 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 Pending 0 20s 2022-03-08T00:31:47.426 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:31:47.479 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:31:47.479 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 Pending 0 30s 2022-03-08T00:31:57.481 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:31:57.541 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:31:57.541 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 Pending 0 40s 2022-03-08T00:32:07.544 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:07.598 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:07.599 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 50s 2022-03-08T00:32:17.600 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:17.952 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:17.953 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 60s 2022-03-08T00:32:27.957 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:28.037 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:28.038 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 71s 2022-03-08T00:32:38.042 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:38.106 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:38.107 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 81s 2022-03-08T00:32:48.109 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:48.177 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:48.178 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 91s 2022-03-08T00:32:58.180 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:32:58.244 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:32:58.245 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 101s 2022-03-08T00:33:08.248 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:33:08.807 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:08.807 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 0/1 ContainerCreating 0 111s 2022-03-08T00:33:18.809 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T00:33:18.876 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:18.876 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-operator-6b496c8959-m26j6 1/1 Running 0 2m1s 2022-03-08T00:33:18.878 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-m26j6 2022-03-08T00:33:18.952 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:33:18.952 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T00:33:18.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.037397 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T00:33:18.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.037486 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-08T00:33:18.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.037489 I | cephcmd: starting Rook-Ceph operator 2022-03-08T00:33:18.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.205654 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-08T00:33:18.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.212392 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T00:33:18.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.214326 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T00:33:18.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.217613 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-08T00:33:18.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.219163 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T00:33:18.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.221509 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T00:33:18.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.221526 I | operator: watching all namespaces for ceph cluster CRs 2022-03-08T00:33:18.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.221621 I | operator: setting up the controller-runtime manager 2022-03-08T00:33:18.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.224690 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-08T00:33:18.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.230534 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T00:33:18.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629054 I | ceph-cluster-controller: successfully started 2022-03-08T00:33:18.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629131 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T00:33:18.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629151 I | ceph-crashcollector-controller: successfully started 2022-03-08T00:33:18.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629210 I | ceph-block-pool-controller: successfully started 2022-03-08T00:33:18.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629277 I | ceph-object-store-user-controller: successfully started 2022-03-08T00:33:18.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629340 I | ceph-object-realm-controller: successfully started 2022-03-08T00:33:18.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629396 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T00:33:18.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629456 I | ceph-object-zone-controller: successfully started 2022-03-08T00:33:18.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629589 I | ceph-object-controller: successfully started 2022-03-08T00:33:18.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629669 I | ceph-file-controller: successfully started 2022-03-08T00:33:18.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629741 I | ceph-nfs-controller: successfully started 2022-03-08T00:33:18.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629810 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T00:33:18.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629880 I | ceph-client-controller: successfully started 2022-03-08T00:33:18.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.629934 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T00:33:18.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:12.631193 I | operator: starting the controller-runtime manager 2022-03-08T00:33:18.977 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-08T00:33:18.977 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T00:33:18.978 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T00:33:18.979 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T00:33:18.979 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T00:33:18.980 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T00:33:18.980 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T00:33:18.980 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T00:33:18.981 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T00:33:18.984 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f - 2022-03-08T00:33:20.000 INFO:teuthology.orchestra.run.smithi120.stdout:configmap/rook-config-override created 2022-03-08T00:33:20.004 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-03-08T00:33:20.005 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:33:20.005 DEBUG:teuthology.orchestra.run.smithi120:> dd of=cluster.yaml 2022-03-08T00:33:20.011 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f cluster.yaml 2022-03-08T00:33:22.449 INFO:teuthology.orchestra.run.smithi120.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T00:33:22.458 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.457219 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-08T00:33:22.458 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.457267 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T00:33:22.466 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.465157 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:33:22.470 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.465701 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T00:33:22.470 INFO:tasks.rook.operator.smithi120.stdout: ... // 3 identical fields 2022-03-08T00:33:22.470 INFO:tasks.rook.operator.smithi120.stdout: Labels: nil, 2022-03-08T00:33:22.471 INFO:tasks.rook.operator.smithi120.stdout: Placement: nil, 2022-03-08T00:33:22.471 INFO:tasks.rook.operator.smithi120.stdout: Network: v1.NetworkSpec{ 2022-03-08T00:33:22.471 INFO:tasks.rook.operator.smithi120.stdout: Provider: "", 2022-03-08T00:33:22.471 INFO:tasks.rook.operator.smithi120.stdout: Selectors: nil, 2022-03-08T00:33:22.471 INFO:tasks.rook.operator.smithi120.stdout: HostNetwork: false, 2022-03-08T00:33:22.472 INFO:tasks.rook.operator.smithi120.stdout:- IPFamily: "", 2022-03-08T00:33:22.473 INFO:tasks.rook.operator.smithi120.stdout:+ IPFamily: "IPv4", 2022-03-08T00:33:22.473 INFO:tasks.rook.operator.smithi120.stdout: DualStack: false, 2022-03-08T00:33:22.473 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:33:22.474 INFO:tasks.rook.operator.smithi120.stdout: Resources: nil, 2022-03-08T00:33:22.474 INFO:tasks.rook.operator.smithi120.stdout: PriorityClassNames: nil, 2022-03-08T00:33:22.474 INFO:tasks.rook.operator.smithi120.stdout: ... // 16 identical fields 2022-03-08T00:33:22.474 INFO:tasks.rook.operator.smithi120.stdout: } 2022-03-08T00:33:22.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.467934 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T00:33:22.476 DEBUG:teuthology.orchestra.remote:smithi120:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T00:33:22.482 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.480940 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T00:33:22.487 DEBUG:teuthology.orchestra.run.smithi120:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T00:33:22.500 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.499273 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-08T00:33:22.501 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.499379 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T00:33:22.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.502466 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T00:33:22.507 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.505547 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T00:33:22.510 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.507748 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T00:33:22.512 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.509760 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T00:33:22.512 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.511446 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-08T00:33:22.555 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.553959 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:33:22.726 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:22.725092 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-08T00:33:22.781 INFO:teuthology.orchestra.run.smithi120.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T00:33:22.784 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T00:33:22.784 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:22.896 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:22.897 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 Pending 0 0s 2022-03-08T00:33:23.087 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:23.085762 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-08T00:33:23.470 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:23.469369 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-08T00:33:23.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:23.753035 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-08T00:33:23.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:23.980611 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T00:33:24.155 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:24.069643 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-08T00:33:24.271 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:24.270431 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T00:33:24.471 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:24.470158 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T00:33:24.472 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:24.470191 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-08T00:33:24.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:33:24.670342 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T00:33:27.900 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:28.004 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:28.004 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 6s 2022-03-08T00:33:33.007 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:33.276 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:33.276 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 11s 2022-03-08T00:33:38.280 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:38.345 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:38.345 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 16s 2022-03-08T00:33:43.347 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:43.410 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:43.410 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 21s 2022-03-08T00:33:48.416 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:48.502 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:48.502 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 26s 2022-03-08T00:33:53.506 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:53.575 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:53.575 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 31s 2022-03-08T00:33:58.577 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:33:58.634 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:33:58.635 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 36s 2022-03-08T00:34:01.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:01.887516 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-08T00:34:01.923 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:01.922070 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T00:34:01.927 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:01.925397 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T00:34:02.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.096840 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T00:34:02.123 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.122034 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T00:34:02.126 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.124751 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T00:34:02.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.554665 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:34:02.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.554713 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:34:02.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.554736 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:34:02.559 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.556880 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T00:34:02.625 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.623853 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:34:02.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.624458 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T00:34:02.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.734216 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T00:34:02.747 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.744480 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T00:34:02.755 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.754920 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T00:34:02.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.783264 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T00:34:02.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.784576 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 00:34:02.755669383 +0000 UTC m=+50.771640062 LastTransitionTime:2022-03-08 00:34:02.755669243 +0000 UTC m=+50.771639957}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:34:02.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:02.959725 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T00:34:03.443 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.439425 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T00:34:03.444 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.439455 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T00:34:03.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.481803 I | op-mon: start running mons 2022-03-08T00:34:03.637 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:03.682 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.681141 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-08T00:34:03.731 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:03.731 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 41s 2022-03-08T00:34:03.817 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.815691 I | op-mon: creating mon secrets for a new cluster 2022-03-08T00:34:03.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:03.963122 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T00:34:04.761 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:04.760035 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T00:34:04.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:04.959392 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T00:34:05.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:05.160494 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T00:34:05.364 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:05.363033 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T00:34:05.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:05.596216 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T00:34:05.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:05.962910 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T00:34:06.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:06.360310 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T00:34:06.761 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:06.760480 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T00:34:06.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:06.960251 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:34:07.006 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:06.960529 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:34:07.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:07.160363 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T00:34:07.561 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:07.559771 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T00:34:08.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:08.047218 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T00:34:08.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:08.360511 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T00:34:08.377 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:08.376640 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T00:34:08.734 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:08.937 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:08.938 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 46s 2022-03-08T00:34:09.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.160765 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T00:34:09.363 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.362539 I | op-mon: targeting the mon count 1 2022-03-08T00:34:09.725 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.722549 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T00:34:09.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.725765 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T00:34:09.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.725920 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:34:09.773 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:09.772312 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T00:34:10.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:10.211944 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T00:34:10.388 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:10.387493 I | op-mon: parsing mon endpoints: 2022-03-08T00:34:10.388 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:10.387513 W | op-mon: ignoring invalid monitor 2022-03-08T00:34:10.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:10.455435 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:34:10.836 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:10.835067 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T00:34:11.114 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:11.112778 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T00:34:11.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:11.280552 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T00:34:12.445 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:12.444548 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T00:34:12.745 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:12.744860 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T00:34:13.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:13.089203 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-08T00:34:13.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:13.464993 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T00:34:13.941 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:14.116 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:14.117 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 52s 2022-03-08T00:34:19.118 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:19.526 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:19.526 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 57s 2022-03-08T00:34:24.366 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.364109 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:34:24.366 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.364127 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:34:24.367 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.364135 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:34:24.441 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.440715 I | op-mon: parsing mon endpoints: 2022-03-08T00:34:24.442 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.440735 W | op-mon: ignoring invalid monitor 2022-03-08T00:34:24.528 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:24.747 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.746492 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:34:24.750 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.747576 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:34:24.751 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:24.752 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 62s 2022-03-08T00:34:24.862 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.860809 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T00:34:24.862 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:34:24.854+0000 7f2993fff700 -1 failed for service _ceph-mon._tcp 2022-03-08T00:34:24.862 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:34:24.854+0000 7f2993fff700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T00:34:24.863 INFO:tasks.rook.operator.smithi120.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-08T00:34:24.863 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:24.860840 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:34:25.365 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.363624 I | op-mon: start running mons 2022-03-08T00:34:25.669 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.668505 I | op-mon: parsing mon endpoints: 2022-03-08T00:34:25.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.668541 W | op-mon: ignoring invalid monitor 2022-03-08T00:34:25.885 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.882736 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T00:34:25.954 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.952908 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:34:25.954 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:25.953094 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:34:26.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:26.468545 I | op-mon: targeting the mon count 1 2022-03-08T00:34:26.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:26.475314 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T00:34:26.863 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:26.862934 I | op-mon: sched-mon: waiting for canary pod creation rook-ceph-mon-a-canary 2022-03-08T00:34:29.753 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:29.808 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:29.809 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 67s 2022-03-08T00:34:33.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:33.076075 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi120 2022-03-08T00:34:33.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:33.076104 I | op-mon: assignmon: mon a assigned to node smithi120 2022-03-08T00:34:33.612 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:33.611228 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T00:34:33.988 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:33.987048 I | op-mon: creating mon a 2022-03-08T00:34:34.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:34.502275 I | op-mon: mon "a" endpoint is [v2:10.98.3.151:3300,v1:10.98.3.151:6789] 2022-03-08T00:34:34.811 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:34.997 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:34.998 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 0/1 ContainerCreating 0 72s 2022-03-08T00:34:35.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.234941 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:-1] 2022-03-08T00:34:35.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.235123 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T00:34:35.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.235222 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T00:34:35.403 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.402599 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:34:35.410 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.409468 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:34:35.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.641503 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T00:34:35.843 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:35.841163 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T00:34:36.094 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:36.092874 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:34:36.094 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:36.092923 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:34:36.101 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:36.099239 I | op-mon: mon a is not yet running 2022-03-08T00:34:36.101 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:36.099270 I | op-mon: mons running: [] 2022-03-08T00:34:40.000 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T00:34:40.226 INFO:teuthology.orchestra.run.smithi120.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T00:34:40.226 INFO:teuthology.orchestra.run.smithi120.stdout:rook-ceph-tools-7677f7f4b9-p278g 1/1 Running 0 78s 2022-03-08T00:34:40.227 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T00:34:40.228 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:34:40.480 INFO:teuthology.orchestra.run.smithi120.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T00:34:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T00:34:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2022-03-08T00:34:40.474+0000 7fcb07fee700 -1 failed for service _ceph-mon._tcpcommand terminated with exit code 1 2022-03-08T00:34:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T00:34:50.485 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:34:53.235 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:34:53.236 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2022-03-08T00:34:53.236 INFO:teuthology.orchestra.run.smithi120.stderr:pg stat 2022-03-08T00:34:53.236 INFO:teuthology.orchestra.run.smithi120.stderr:pg getmap 2022-03-08T00:34:53.236 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T00:34:53.237 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T00:34:53.237 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_pools_json 2022-03-08T00:34:53.237 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-pool [...] 2022-03-08T00:34:53.237 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-primary [] [...] 2022-03-08T00:34:53.237 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-osd [] [...] 2022-03-08T00:34:53.238 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls [] [...] 2022-03-08T00:34:53.238 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T00:34:53.238 INFO:teuthology.orchestra.run.smithi120.stderr:Error EINVAL: invalid command 2022-03-08T00:34:53.239 INFO:teuthology.orchestra.run.smithi120.stderr:command terminated with exit code 22 2022-03-08T00:34:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T00:34:56.212 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.211038 I | op-mon: mons running: [a] 2022-03-08T00:34:56.430 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.429818 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:34:56.431 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.429844 I | op-mon: mons created: 1 2022-03-08T00:34:56.675 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.675056 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:34:56.681 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.681167 I | op-mon: mons running: [a] 2022-03-08T00:34:56.898 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.897395 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:34:56.898 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:56.897437 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:34:57.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.172049 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:34:57.174 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.172073 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:34:57.452 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.451425 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:34:57.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.451458 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:34:57.774 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.773195 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:34:57.774 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:57.773217 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:34:58.055 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.054747 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:34:58.056 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.054773 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:34:58.327 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.326448 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:34:58.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.326474 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T00:34:58.537 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.536265 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T00:34:58.537 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.536308 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T00:34:58.821 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:58.820812 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T00:34:59.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:59.105372 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T00:34:59.405 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:59.403995 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T00:34:59.758 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:59.757330 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T00:34:59.758 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:34:59.757356 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T00:35:00.056 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.055584 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T00:35:00.268 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.267055 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T00:35:00.269 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.267088 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T00:35:00.269 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.267104 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T00:35:00.549 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.548153 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T00:35:00.565 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.564188 I | op-mgr: start running mgr 2022-03-08T00:35:00.565 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:00.564230 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T00:35:03.242 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:35:03.683 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:03.683 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2022-03-08T00:35:03.684 INFO:teuthology.orchestra.run.smithi120.stderr:pg stat 2022-03-08T00:35:03.684 INFO:teuthology.orchestra.run.smithi120.stderr:pg getmap 2022-03-08T00:35:03.684 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T00:35:03.684 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T00:35:03.685 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_pools_json 2022-03-08T00:35:03.685 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-pool [...] 2022-03-08T00:35:03.685 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-primary [] [...] 2022-03-08T00:35:03.685 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-osd [] [...] 2022-03-08T00:35:03.686 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls [] [...] 2022-03-08T00:35:03.686 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T00:35:03.686 INFO:teuthology.orchestra.run.smithi120.stderr:Error EINVAL: invalid command 2022-03-08T00:35:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T00:35:03.688 INFO:teuthology.orchestra.run.smithi120.stderr:command terminated with exit code 22 2022-03-08T00:35:13.138 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.137292 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T00:35:13.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.161040 I | op-mgr: setting services to point to mgr "a" 2022-03-08T00:35:13.224 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.223100 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T00:35:13.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.223129 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T00:35:13.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.223139 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T00:35:13.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.247792 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T00:35:13.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.251620 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T00:35:13.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.251646 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T00:35:13.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.251652 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T00:35:13.611 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.609383 I | op-mgr: successful modules: prometheus 2022-03-08T00:35:13.634 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.634001 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-08T00:35:13.652 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.652152 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T00:35:13.653 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.652171 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:35:13.681 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.680660 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T00:35:13.682 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.680687 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T00:35:13.682 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.680701 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T00:35:13.689 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:35:13.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.724749 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T00:35:13.730 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.724787 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T00:35:13.730 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.726599 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-08T00:35:13.730 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.726746 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:35:13.730 INFO:tasks.rook.operator.smithi120.stdout:I0308 00:35:13.726997 8 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T00:35:13.841 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.841321 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:13.859 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:13.857919 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:35:14.273 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:14.274 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2022-03-08T00:35:14.274 INFO:teuthology.orchestra.run.smithi120.stderr:pg stat 2022-03-08T00:35:14.274 INFO:teuthology.orchestra.run.smithi120.stderr:pg getmap 2022-03-08T00:35:14.274 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T00:35:14.275 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T00:35:14.275 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_pools_json 2022-03-08T00:35:14.275 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-pool [...] 2022-03-08T00:35:14.275 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-primary [] [...] 2022-03-08T00:35:14.276 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-osd [] [...] 2022-03-08T00:35:14.276 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls [] [...] 2022-03-08T00:35:14.276 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T00:35:14.276 INFO:teuthology.orchestra.run.smithi120.stderr:Error EINVAL: invalid command 2022-03-08T00:35:14.284 INFO:teuthology.orchestra.run.smithi120.stderr:command terminated with exit code 22 2022-03-08T00:35:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T00:35:14.356 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:14.355510 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T00:35:14.886 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:14.885785 I | op-mgr: successful modules: dashboard 2022-03-08T00:35:19.669 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:19.668758 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T00:35:19.850 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:19.849134 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T00:35:19.901 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:19.900017 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:35:19.901 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:19.900043 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:35:19.902 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:19.900054 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:35:20.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.287484 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:20.319 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.318270 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:35:20.319 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.318468 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:35:20.536 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.535092 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:35:20.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.628267 I | op-mon: start running mons 2022-03-08T00:35:20.656 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.655419 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:20.676 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.676189 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:35:20.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.683080 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:35:20.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:20.683209 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:35:21.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:21.325965 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T00:35:21.871 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:21.870822 I | op-mon: targeting the mon count 1 2022-03-08T00:35:21.876 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:21.874601 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:35:22.136 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.135794 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:35:22.137 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.135822 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:35:22.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.360915 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:35:22.362 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.360943 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:35:22.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.593728 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:35:22.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.593752 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:35:22.816 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.815224 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:35:22.816 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:22.815256 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:35:23.030 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.029396 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:35:23.030 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.029425 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T00:35:23.250 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.249621 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T00:35:23.250 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.249648 I | op-mon: checking for basic quorum with existing mons 2022-03-08T00:35:23.457 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.456961 I | op-mon: mon "a" endpoint is [v2:10.98.3.151:3300,v1:10.98.3.151:6789] 2022-03-08T00:35:23.479 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.476939 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:35:23.881 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.876657 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:35:23.881 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:23.876950 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:35:24.251 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.250432 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T00:35:24.258 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.258335 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T00:35:24.259 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.258360 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:35:24.287 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:35:24.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.486502 I | op-mon: mons running: [a] 2022-03-08T00:35:24.724 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.723566 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:35:24.724 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.723595 I | op-mon: mons created: 1 2022-03-08T00:35:24.745 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:24.746 INFO:teuthology.orchestra.run.smithi120.stderr:no valid command found; 10 closest matches: 2022-03-08T00:35:24.746 INFO:teuthology.orchestra.run.smithi120.stderr:pg stat 2022-03-08T00:35:24.746 INFO:teuthology.orchestra.run.smithi120.stderr:pg getmap 2022-03-08T00:35:24.746 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T00:35:24.747 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T00:35:24.747 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_pools_json 2022-03-08T00:35:24.747 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-pool [...] 2022-03-08T00:35:24.747 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-primary [] [...] 2022-03-08T00:35:24.748 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls-by-osd [] [...] 2022-03-08T00:35:24.748 INFO:teuthology.orchestra.run.smithi120.stderr:pg ls [] [...] 2022-03-08T00:35:24.748 INFO:teuthology.orchestra.run.smithi120.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T00:35:24.748 INFO:teuthology.orchestra.run.smithi120.stderr:Error EINVAL: invalid command 2022-03-08T00:35:24.750 INFO:teuthology.orchestra.run.smithi120.stderr:command terminated with exit code 22 2022-03-08T00:35:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T00:35:24.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.970822 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:35:24.982 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:24.981686 I | op-mon: mons running: [a] 2022-03-08T00:35:25.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.000886 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T00:35:25.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.205277 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:35:25.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.205304 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T00:35:25.461 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.456158 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T00:35:25.733 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.732517 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T00:35:25.959 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:25.958765 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T00:35:26.243 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.241070 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T00:35:26.243 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.241097 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T00:35:26.632 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.631946 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T00:35:26.641 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.640526 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T00:35:26.879 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.878431 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T00:35:26.880 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.878465 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T00:35:26.880 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:26.878473 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T00:35:27.102 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.101653 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T00:35:27.124 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.123652 I | op-mgr: start running mgr 2022-03-08T00:35:27.124 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.123693 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T00:35:27.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.669671 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T00:35:27.678 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.677892 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T00:35:27.681 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:27.680404 I | op-mgr: setting services to point to mgr "a" 2022-03-08T00:35:28.468 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.467819 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T00:35:28.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.467844 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T00:35:28.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.467854 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T00:35:28.489 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.488927 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T00:35:28.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.654627 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T00:35:28.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.654652 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T00:35:28.656 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.654657 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T00:35:28.865 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:28.864931 I | op-mgr: successful modules: prometheus 2022-03-08T00:35:29.223 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:29.223145 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T00:35:29.260 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:29.259001 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T00:35:29.260 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:29.259031 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:35:29.941 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:29.940528 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T00:35:29.943 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:29.941122 I | op-mgr: successful modules: dashboard 2022-03-08T00:35:30.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:30.289139 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T00:35:31.292 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:31.291606 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T00:35:31.932 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:31.931530 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T00:35:34.634 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:34.633970 I | op-mgr: successful modules: balancer 2022-03-08T00:35:34.753 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:35:35.263 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:35.264 INFO:teuthology.orchestra.run.smithi120.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-08T00:35:35.269 INFO:teuthology.orchestra.run.smithi120.stderr:command terminated with exit code 2 2022-03-08T00:35:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-08T00:35:35.633 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:35.632942 I | op-mgr: successful modules: balancer 2022-03-08T00:35:36.733 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:36.732809 I | cephclient: action set rook backend succeeded on attempt 1 2022-03-08T00:35:36.734 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:36.732834 I | op-mgr: successful modules: orchestrator modules 2022-03-08T00:35:37.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:37.279706 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-08T00:35:37.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:37.279732 I | op-mgr: successful modules: orchestrator modules 2022-03-08T00:35:45.273 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status -f json 2022-03-08T00:35:45.849 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:45.850 INFO:teuthology.orchestra.run.smithi120.stdout:{"available": true, "backend": "rook"} 2022-03-08T00:35:45.871 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T00:35:45.873 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T00:35:46.466 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch apply osd --all-available-devices 2022-03-08T00:35:47.100 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:47.099179 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T00:35:47.100 INFO:tasks.rook.operator.smithi120.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T00:35:47.100 INFO:tasks.rook.operator.smithi120.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T00:35:47.101 INFO:tasks.rook.operator.smithi120.stdout: ... // 3 identical fields 2022-03-08T00:35:47.101 INFO:teuthology.orchestra.run.smithi120.stdout:['Success'] 2022-03-08T00:35:47.103 INFO:tasks.rook.operator.smithi120.stdout: Config: nil, 2022-03-08T00:35:47.103 INFO:tasks.rook.operator.smithi120.stdout: Selection: {}, 2022-03-08T00:35:47.103 INFO:tasks.rook.operator.smithi120.stdout:- StorageClassDeviceSets: nil, 2022-03-08T00:35:47.103 INFO:tasks.rook.operator.smithi120.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T00:35:47.104 INFO:tasks.rook.operator.smithi120.stdout:+ { 2022-03-08T00:35:47.104 INFO:tasks.rook.operator.smithi120.stdout:+ Name: "smithi120-lv-1", 2022-03-08T00:35:47.104 INFO:tasks.rook.operator.smithi120.stdout:+ Count: 1, 2022-03-08T00:35:47.104 INFO:tasks.rook.operator.smithi120.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T00:35:47.105 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:35:47.105 INFO:tasks.rook.operator.smithi120.stdout:+ { 2022-03-08T00:35:47.105 INFO:tasks.rook.operator.smithi120.stdout:+ Name: "smithi120-lv-2", 2022-03-08T00:35:47.105 INFO:tasks.rook.operator.smithi120.stdout:+ Count: 1, 2022-03-08T00:35:47.106 INFO:tasks.rook.operator.smithi120.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T00:35:47.107 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:35:47.107 INFO:tasks.rook.operator.smithi120.stdout:+ { 2022-03-08T00:35:47.107 INFO:tasks.rook.operator.smithi120.stdout:+ Name: "smithi120-lv-3", 2022-03-08T00:35:47.107 INFO:tasks.rook.operator.smithi120.stdout:+ Count: 1, 2022-03-08T00:35:47.108 INFO:tasks.rook.operator.smithi120.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T00:35:47.108 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:35:47.108 INFO:tasks.rook.operator.smithi120.stdout:+ { 2022-03-08T00:35:47.109 INFO:tasks.rook.operator.smithi120.stdout:+ Name: "smithi120-lv-4", 2022-03-08T00:35:47.109 INFO:tasks.rook.operator.smithi120.stdout:+ Count: 1, 2022-03-08T00:35:47.109 INFO:tasks.rook.operator.smithi120.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T00:35:47.110 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:35:47.110 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:35:47.110 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:35:47.110 INFO:tasks.rook.operator.smithi120.stdout: Annotations: nil, 2022-03-08T00:35:47.111 INFO:tasks.rook.operator.smithi120.stdout: Labels: nil, 2022-03-08T00:35:47.111 INFO:tasks.rook.operator.smithi120.stdout: ... // 20 identical fields 2022-03-08T00:35:47.111 INFO:tasks.rook.operator.smithi120.stdout: } 2022-03-08T00:35:47.111 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:47.099273 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:35:47.121 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-08T00:35:47.122 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:35:47.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:47.123992 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:47.141 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:47.140839 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:35:47.647 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:47.664 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T00:35:47.665 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:35:53.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.593663 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:35:53.595 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.593695 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:35:53.595 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.593709 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:35:53.734 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.734305 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:53.812 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.811844 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:35:53.815 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:53.812204 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:35:54.241 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.240902 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:35:54.387 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.386932 I | op-mon: start running mons 2022-03-08T00:35:54.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.514287 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:54.564 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.564276 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:35:54.582 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.572550 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:35:54.582 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:54.572780 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:35:55.344 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:55.344140 I | op-mon: targeting the mon count 1 2022-03-08T00:35:55.356 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:55.355216 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T00:35:56.356 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:56.356086 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:35:56.362 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:56.362363 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:35:56.377 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:56.376814 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:35:56.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:56.421317 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T00:35:56.426 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:56.426577 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T00:35:57.667 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:35:58.192 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:35:58.213 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T00:35:58.213 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:35:58.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:35:58.433422 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T00:36:04.454 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.453658 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:36:04.454 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.453681 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:36:04.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.453690 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:36:04.495 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.495028 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:36:04.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.497770 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:36:04.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.499730 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:36:04.892 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:04.891199 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:36:05.024 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:05.023648 I | op-mon: start running mons 2022-03-08T00:36:05.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:05.106175 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:36:05.123 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:05.122702 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:36:05.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:05.169362 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:36:05.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:05.169736 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:36:06.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:06.504079 I | op-mon: targeting the mon count 1 2022-03-08T00:36:06.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:06.507718 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:36:06.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:06.887417 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:36:06.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:06.887448 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:36:07.226 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.225466 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:36:07.226 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.225498 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:36:07.564 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.563937 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:36:07.565 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.563959 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:36:07.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.899918 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:36:07.919 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:07.899947 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:36:08.214 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:36:08.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.237358 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:36:08.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.237387 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T00:36:08.616 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.616160 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T00:36:08.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.616193 I | op-mon: checking for basic quorum with existing mons 2022-03-08T00:36:08.716 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:36:08.734 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T00:36:08.735 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:36:08.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.850736 I | op-mon: mon "a" endpoint is [v2:10.98.3.151:3300,v1:10.98.3.151:6789] 2022-03-08T00:36:08.962 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.961850 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:36:08.975 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.975118 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:36:08.976 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:08.975258 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:36:09.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.031249 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T00:36:09.042 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.041703 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T00:36:09.042 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.041988 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:36:09.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.224667 I | op-mon: mons running: [a] 2022-03-08T00:36:09.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.653468 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:36:09.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:09.653495 I | op-mon: mons created: 1 2022-03-08T00:36:10.057 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:10.056751 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:36:10.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:10.063923 I | op-mon: mons running: [a] 2022-03-08T00:36:10.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:10.473776 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:36:10.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:10.473816 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T00:36:10.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:10.900278 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T00:36:11.337 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:11.335688 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T00:36:11.752 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:11.751910 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T00:36:12.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:12.210569 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T00:36:12.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:12.210590 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T00:36:12.639 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:12.638821 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T00:36:13.044 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.044336 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T00:36:13.045 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.044361 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T00:36:13.045 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.044365 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T00:36:13.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.508014 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T00:36:13.532 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.531764 I | op-mgr: start running mgr 2022-03-08T00:36:13.532 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:13.531789 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T00:36:14.011 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.011599 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T00:36:14.019 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.019147 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T00:36:14.022 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.021469 I | op-mgr: setting services to point to mgr "a" 2022-03-08T00:36:14.168 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.168095 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T00:36:14.172 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.170031 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T00:36:14.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.198269 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T00:36:14.202 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.200564 I | op-osd: creating 1 new PVCs for device set "smithi120-lv-1" 2022-03-08T00:36:14.401 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.401433 I | op-osd: successfully provisioned PVC "smithi120-lv-1-data-0r7tlx" 2022-03-08T00:36:14.402 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.401466 I | op-osd: creating 1 new PVCs for device set "smithi120-lv-2" 2022-03-08T00:36:14.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.580883 I | op-osd: successfully provisioned PVC "smithi120-lv-2-data-0twdg6" 2022-03-08T00:36:14.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.580913 I | op-osd: creating 1 new PVCs for device set "smithi120-lv-3" 2022-03-08T00:36:14.780 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.780097 I | op-osd: successfully provisioned PVC "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:36:14.781 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.780145 I | op-osd: creating 1 new PVCs for device set "smithi120-lv-4" 2022-03-08T00:36:14.982 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.981005 I | op-osd: successfully provisioned PVC "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:36:14.988 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:14.988697 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0r7tlx" 2022-03-08T00:36:15.435 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.435253 I | op-mgr: successful modules: dashboard 2022-03-08T00:36:15.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.435316 I | op-mgr: successful modules: prometheus 2022-03-08T00:36:15.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.435501 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T00:36:15.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.592748 I | op-osd: started OSD provisioning job for PVC "smithi120-lv-1-data-0r7tlx" 2022-03-08T00:36:15.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.592768 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:36:15.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:15.893683 I | op-mgr: successful modules: balancer 2022-03-08T00:36:16.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:16.093448 I | op-osd: started OSD provisioning job for PVC "smithi120-lv-2-data-0twdg6" 2022-03-08T00:36:16.094 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:16.093482 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:36:16.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:16.455417 I | op-osd: started OSD provisioning job for PVC "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:36:16.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:16.455450 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:36:17.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.520925 I | op-osd: started OSD provisioning job for PVC "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:36:17.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.520954 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T00:36:17.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.520961 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T00:36:17.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.641913 I | op-mgr: successful modules: orchestrator modules 2022-03-08T00:36:17.645 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.643942 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0r7tlx is "starting" 2022-03-08T00:36:17.645 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.643961 I | op-osd: OSD orchestration status for PVC smithi120-lv-2-data-0twdg6 is "starting" 2022-03-08T00:36:17.645 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.643966 I | op-osd: OSD orchestration status for PVC smithi120-lv-3-data-0q4xnx is "starting" 2022-03-08T00:36:17.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:17.643971 I | op-osd: OSD orchestration status for PVC smithi120-lv-4-data-0vbrdz is "starting" 2022-03-08T00:36:18.736 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:36:19.787 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:36:19.804 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T00:36:19.804 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:36:29.805 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:36:31.193 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:36:31.212 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T00:36:31.212 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:36:35.049 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:35.049031 I | op-osd: OSD orchestration status for node smithi120-lv-1-data-0r7tlx is "orchestrating" 2022-03-08T00:36:35.358 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:35.358465 I | op-osd: OSD orchestration status for node smithi120-lv-2-data-0twdg6 is "orchestrating" 2022-03-08T00:36:35.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:35.398361 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0r7tlx is "orchestrating" 2022-03-08T00:36:36.032 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:36.031894 I | op-osd: OSD orchestration status for PVC smithi120-lv-2-data-0twdg6 is "orchestrating" 2022-03-08T00:36:37.845 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:37.845180 I | op-osd: OSD orchestration status for node smithi120-lv-3-data-0q4xnx is "orchestrating" 2022-03-08T00:36:38.501 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:38.500939 I | op-osd: OSD orchestration status for PVC smithi120-lv-3-data-0q4xnx is "orchestrating" 2022-03-08T00:36:38.699 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:38.699180 I | op-osd: OSD orchestration status for node smithi120-lv-4-data-0vbrdz is "orchestrating" 2022-03-08T00:36:39.085 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:39.085420 I | op-osd: OSD orchestration status for PVC smithi120-lv-4-data-0vbrdz is "orchestrating" 2022-03-08T00:36:41.213 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:36:41.748 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:36:41.766 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:36:41.767 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:36:41.995 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:41.995000 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0r7tlx is "completed" 2022-03-08T00:36:41.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:41.995038 I | op-osd: creating OSD 0 on PVC "smithi120-lv-1-data-0r7tlx" 2022-03-08T00:36:41.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:41.995053 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0r7tlx" 2022-03-08T00:36:43.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:43.172936 I | op-osd: OSD orchestration status for PVC smithi120-lv-2-data-0twdg6 is "completed" 2022-03-08T00:36:43.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:43.172965 I | op-osd: creating OSD 1 on PVC "smithi120-lv-2-data-0twdg6" 2022-03-08T00:36:43.174 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:43.172978 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:36:44.429 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:44.429051 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi120": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:36:44.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:44.974882 I | op-osd: OSD orchestration status for PVC smithi120-lv-3-data-0q4xnx is "completed" 2022-03-08T00:36:44.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:44.975347 I | op-osd: creating OSD 2 on PVC "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:36:44.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:44.976727 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:36:46.830 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:46.829646 I | op-osd: OSD orchestration status for PVC smithi120-lv-4-data-0vbrdz is "completed" 2022-03-08T00:36:46.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:46.830259 I | op-osd: creating OSD 3 on PVC "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:36:46.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:46.830321 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:36:48.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:48.201196 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T00:36:48.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:36:48.201214 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:36:51.768 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:36:53.388 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:36:53.407 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:36:53.408 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:37:03.409 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:37:04.186 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:04.204 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:37:04.204 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:37:14.205 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:37:15.373 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:15.390 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:37:15.390 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T00:37:25.392 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:37:25.927 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:25.951 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":15,"num_osds":4,"num_up_osds":1,"osd_up_since":1646699843,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:37:25.951 INFO:tasks.rook: have 1/4 OSDs 2022-03-08T00:37:35.953 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd stat -f json 2022-03-08T00:37:36.509 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:36.530 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":22,"num_osds":4,"num_up_osds":4,"osd_up_since":1646699848,"num_in_osds":4,"osd_in_since":1646699800,"num_remapped_pgs":0} 2022-03-08T00:37:36.531 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T00:37:36.531 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- cat /etc/ceph/ceph.conf 2022-03-08T00:37:36.738 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2022-03-08T00:37:36.738 INFO:teuthology.orchestra.run.smithi120.stdout:mon_host = 10.98.3.151:6789 2022-03-08T00:37:36.739 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:36.739 INFO:teuthology.orchestra.run.smithi120.stdout:[client.admin] 2022-03-08T00:37:36.740 INFO:teuthology.orchestra.run.smithi120.stdout:keyring = /etc/ceph/keyring 2022-03-08T00:37:36.742 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- cat /etc/ceph/keyring 2022-03-08T00:37:36.946 INFO:teuthology.orchestra.run.smithi120.stdout:[client.admin] 2022-03-08T00:37:36.946 INFO:teuthology.orchestra.run.smithi120.stdout:key = AQB7pCZiFeGHMBAAaCGJnaLMtOOzJcDa1ygKrA== 2022-03-08T00:37:36.950 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2022-03-08T00:37:36.980 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:37:36.981 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T00:37:37.056 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:37:37.057 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/keyring 2022-03-08T00:37:37.121 INFO:tasks.rook:Setting up client nodes... 2022-03-08T00:37:37.123 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T00:37:37.793 INFO:teuthology.orchestra.run.smithi120.stdout:[client.a] 2022-03-08T00:37:37.793 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBRpSZiLvkCLBAAu2CkKN+zEIGqSBLzghL5gg== 2022-03-08T00:37:37.815 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:37:37.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T00:37:37.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T00:37:37.836 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd dump --format=json 2022-03-08T00:37:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:38.331 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":22,"fsid":"034a23aa-8c1d-44a2-ad04-5ada309e2217","created":"2022-03-08T00:34:51.786065+0000","modified":"2022-03-08T00:37:33.938701+0000","last_up_change":"2022-03-08T00:37:28.904528+0000","last_in_change":"2022-03-08T00:36:40.522292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T00:37:30.735735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9fd64977-8dd9-481b-aaa0-4be8644878c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6800","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6801","nonce":2418932324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6802","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6803","nonce":2418932324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6806","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6807","nonce":2418932324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6804","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6805","nonce":2418932324}]},"public_addr":"10.251.189.22:6801/2418932324","cluster_addr":"10.251.189.22:6803/2418932324","heartbeat_back_addr":"10.251.189.22:6807/2418932324","heartbeat_front_addr":"10.251.189.22:6805/2418932324","state":["exists","up"]},{"osd":1,"uuid":"e57e5c50-8458-4829-a3bf-da1a66169840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6800","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6801","nonce":1124804360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6802","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6803","nonce":1124804360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6806","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6807","nonce":1124804360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6804","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6805","nonce":1124804360}]},"public_addr":"10.251.189.24:6801/1124804360","cluster_addr":"10.251.189.24:6803/1124804360","heartbeat_back_addr":"10.251.189.24:6807/1124804360","heartbeat_front_addr":"10.251.189.24:6805/1124804360","state":["exists","up"]},{"osd":2,"uuid":"a4e5348d-e4cd-40f4-b417-6e5cc50ab658","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6800","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6801","nonce":3274209485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6802","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6803","nonce":3274209485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6806","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6807","nonce":3274209485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6804","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6805","nonce":3274209485}]},"public_addr":"10.251.189.25:6801/3274209485","cluster_addr":"10.251.189.25:6803/3274209485","heartbeat_back_addr":"10.251.189.25:6807/3274209485","heartbeat_front_addr":"10.251.189.25:6805/3274209485","state":["exists","up"]},{"osd":3,"uuid":"742f0c11-49b1-4af5-b8a6-6e177ec47beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6800","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6801","nonce":2765646656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6802","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6803","nonce":2765646656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6806","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6807","nonce":2765646656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6804","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6805","nonce":2765646656}]},"public_addr":"10.251.189.26:6801/2765646656","cluster_addr":"10.251.189.26:6803/2765646656","heartbeat_back_addr":"10.251.189.26:6807/2765646656","heartbeat_front_addr":"10.251.189.26:6805/2765646656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:21.773730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.202835+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.338091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:25.699078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T00:37:38.355 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T00:37:30.735735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T00:37:38.355 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd pool get .mgr pg_num 2022-03-08T00:37:38.857 INFO:teuthology.orchestra.run.smithi120.stdout:pg_num: 1 2022-03-08T00:37:38.879 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T00:37:38.880 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T00:37:38.881 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph mgr dump --format=json 2022-03-08T00:37:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:39.444 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":16,"active_gid":4181,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.13:6800","nonce":2429357606},{"type":"v1","addr":"10.251.189.13:6801","nonce":2429357606}]},"active_addr":"10.251.189.13:6801/2429357606","active_change":"2022-03-08T00:35:31.466303+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.251.189.13:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.251.189.13:0","nonce":3003805526}]},{"addrvec":[{"type":"v2","addr":"10.251.189.13:0","nonce":2731167830}]},{"addrvec":[{"type":"v2","addr":"10.251.189.13:0","nonce":269248555}]},{"addrvec":[{"type":"v2","addr":"10.251.189.13:0","nonce":2596434357}]}]}} 2022-03-08T00:37:39.447 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T00:37:39.447 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T00:37:39.448 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd dump --format=json 2022-03-08T00:37:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:39.946 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":22,"fsid":"034a23aa-8c1d-44a2-ad04-5ada309e2217","created":"2022-03-08T00:34:51.786065+0000","modified":"2022-03-08T00:37:33.938701+0000","last_up_change":"2022-03-08T00:37:28.904528+0000","last_in_change":"2022-03-08T00:36:40.522292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T00:37:30.735735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9fd64977-8dd9-481b-aaa0-4be8644878c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6800","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6801","nonce":2418932324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6802","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6803","nonce":2418932324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6806","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6807","nonce":2418932324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6804","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6805","nonce":2418932324}]},"public_addr":"10.251.189.22:6801/2418932324","cluster_addr":"10.251.189.22:6803/2418932324","heartbeat_back_addr":"10.251.189.22:6807/2418932324","heartbeat_front_addr":"10.251.189.22:6805/2418932324","state":["exists","up"]},{"osd":1,"uuid":"e57e5c50-8458-4829-a3bf-da1a66169840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6800","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6801","nonce":1124804360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6802","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6803","nonce":1124804360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6806","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6807","nonce":1124804360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6804","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6805","nonce":1124804360}]},"public_addr":"10.251.189.24:6801/1124804360","cluster_addr":"10.251.189.24:6803/1124804360","heartbeat_back_addr":"10.251.189.24:6807/1124804360","heartbeat_front_addr":"10.251.189.24:6805/1124804360","state":["exists","up"]},{"osd":2,"uuid":"a4e5348d-e4cd-40f4-b417-6e5cc50ab658","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6800","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6801","nonce":3274209485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6802","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6803","nonce":3274209485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6806","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6807","nonce":3274209485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6804","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6805","nonce":3274209485}]},"public_addr":"10.251.189.25:6801/3274209485","cluster_addr":"10.251.189.25:6803/3274209485","heartbeat_back_addr":"10.251.189.25:6807/3274209485","heartbeat_front_addr":"10.251.189.25:6805/3274209485","state":["exists","up"]},{"osd":3,"uuid":"742f0c11-49b1-4af5-b8a6-6e177ec47beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6800","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6801","nonce":2765646656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6802","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6803","nonce":2765646656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6806","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6807","nonce":2765646656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6804","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6805","nonce":2765646656}]},"public_addr":"10.251.189.26:6801/2765646656","cluster_addr":"10.251.189.26:6803/2765646656","heartbeat_back_addr":"10.251.189.26:6807/2765646656","heartbeat_front_addr":"10.251.189.26:6805/2765646656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:21.773730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.202835+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.338091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:25.699078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T00:37:39.967 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T00:37:39.967 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd dump --format=json 2022-03-08T00:37:40.476 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:40.477 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":22,"fsid":"034a23aa-8c1d-44a2-ad04-5ada309e2217","created":"2022-03-08T00:34:51.786065+0000","modified":"2022-03-08T00:37:33.938701+0000","last_up_change":"2022-03-08T00:37:28.904528+0000","last_in_change":"2022-03-08T00:36:40.522292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T00:37:30.735735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9fd64977-8dd9-481b-aaa0-4be8644878c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6800","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6801","nonce":2418932324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6802","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6803","nonce":2418932324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6806","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6807","nonce":2418932324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.22:6804","nonce":2418932324},{"type":"v1","addr":"10.251.189.22:6805","nonce":2418932324}]},"public_addr":"10.251.189.22:6801/2418932324","cluster_addr":"10.251.189.22:6803/2418932324","heartbeat_back_addr":"10.251.189.22:6807/2418932324","heartbeat_front_addr":"10.251.189.22:6805/2418932324","state":["exists","up"]},{"osd":1,"uuid":"e57e5c50-8458-4829-a3bf-da1a66169840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6800","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6801","nonce":1124804360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6802","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6803","nonce":1124804360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6806","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6807","nonce":1124804360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.24:6804","nonce":1124804360},{"type":"v1","addr":"10.251.189.24:6805","nonce":1124804360}]},"public_addr":"10.251.189.24:6801/1124804360","cluster_addr":"10.251.189.24:6803/1124804360","heartbeat_back_addr":"10.251.189.24:6807/1124804360","heartbeat_front_addr":"10.251.189.24:6805/1124804360","state":["exists","up"]},{"osd":2,"uuid":"a4e5348d-e4cd-40f4-b417-6e5cc50ab658","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6800","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6801","nonce":3274209485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6802","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6803","nonce":3274209485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6806","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6807","nonce":3274209485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.25:6804","nonce":3274209485},{"type":"v1","addr":"10.251.189.25:6805","nonce":3274209485}]},"public_addr":"10.251.189.25:6801/3274209485","cluster_addr":"10.251.189.25:6803/3274209485","heartbeat_back_addr":"10.251.189.25:6807/3274209485","heartbeat_front_addr":"10.251.189.25:6805/3274209485","state":["exists","up"]},{"osd":3,"uuid":"742f0c11-49b1-4af5-b8a6-6e177ec47beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6800","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6801","nonce":2765646656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6802","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6803","nonce":2765646656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6806","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6807","nonce":2765646656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.189.26:6804","nonce":2765646656},{"type":"v1","addr":"10.251.189.26:6805","nonce":2765646656}]},"public_addr":"10.251.189.26:6801/2765646656","cluster_addr":"10.251.189.26:6803/2765646656","heartbeat_back_addr":"10.251.189.26:6807/2765646656","heartbeat_front_addr":"10.251.189.26:6805/2765646656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:21.773730+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.202835+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:27.338091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T00:37:25.699078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T00:37:40.495 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph tell osd.0 flush_pg_stats 2022-03-08T00:37:40.496 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph tell osd.1 flush_pg_stats 2022-03-08T00:37:40.496 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph tell osd.2 flush_pg_stats 2022-03-08T00:37:40.497 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph tell osd.3 flush_pg_stats 2022-03-08T00:37:40.845 INFO:teuthology.orchestra.run.smithi120.stdout:77309411334 2022-03-08T00:37:40.846 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.1 2022-03-08T00:37:40.998 INFO:teuthology.orchestra.run.smithi120.stdout:73014444036 2022-03-08T00:37:40.999 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.3 2022-03-08T00:37:41.002 INFO:teuthology.orchestra.run.smithi120.stdout:55834574853 2022-03-08T00:37:41.003 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.0 2022-03-08T00:37:41.075 INFO:teuthology.orchestra.run.smithi120.stdout:77309411333 2022-03-08T00:37:41.076 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.2 2022-03-08T00:37:41.409 INFO:teuthology.orchestra.run.smithi120.stdout:77309411333 2022-03-08T00:37:41.434 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411334 got 77309411333 for osd.1 2022-03-08T00:37:41.529 INFO:teuthology.orchestra.run.smithi120.stdout:73014444035 2022-03-08T00:37:41.554 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444036 got 73014444035 for osd.3 2022-03-08T00:37:41.560 INFO:teuthology.orchestra.run.smithi120.stdout:55834574852 2022-03-08T00:37:41.578 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2022-03-08T00:37:41.626 INFO:teuthology.orchestra.run.smithi120.stdout:77309411332 2022-03-08T00:37:41.645 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2022-03-08T00:37:42.436 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.1 2022-03-08T00:37:42.555 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.3 2022-03-08T00:37:42.580 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.0 2022-03-08T00:37:42.646 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.2 2022-03-08T00:37:43.073 INFO:teuthology.orchestra.run.smithi120.stdout:73014444035 2022-03-08T00:37:43.091 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444036 got 73014444035 for osd.3 2022-03-08T00:37:43.091 INFO:teuthology.orchestra.run.smithi120.stdout:77309411333 2022-03-08T00:37:43.113 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411334 got 77309411333 for osd.1 2022-03-08T00:37:43.143 INFO:teuthology.orchestra.run.smithi120.stdout:55834574852 2022-03-08T00:37:43.163 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2022-03-08T00:37:43.291 INFO:teuthology.orchestra.run.smithi120.stdout:77309411333 2022-03-08T00:37:43.312 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.2 2022-03-08T00:37:43.313 DEBUG:teuthology.parallel:result is None 2022-03-08T00:37:44.092 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.3 2022-03-08T00:37:44.114 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.1 2022-03-08T00:37:44.165 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd last-stat-seq osd.0 2022-03-08T00:37:44.618 INFO:teuthology.orchestra.run.smithi120.stdout:73014444037 2022-03-08T00:37:44.635 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444036 got 73014444037 for osd.3 2022-03-08T00:37:44.635 DEBUG:teuthology.parallel:result is None 2022-03-08T00:37:44.697 INFO:teuthology.orchestra.run.smithi120.stdout:77309411334 2022-03-08T00:37:44.716 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411334 got 77309411334 for osd.1 2022-03-08T00:37:44.717 DEBUG:teuthology.parallel:result is None 2022-03-08T00:37:44.728 INFO:teuthology.orchestra.run.smithi120.stdout:55834574854 2022-03-08T00:37:44.748 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2022-03-08T00:37:44.748 DEBUG:teuthology.parallel:result is None 2022-03-08T00:37:44.749 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T00:37:44.749 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph pg dump --format=json 2022-03-08T00:37:45.245 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:45.248 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2022-03-08T00:37:45.264 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-08T00:37:44.199514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23076,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":374957532,"statfs":{"total":383980142592,"available":383956512768,"internally_reserved":0,"allocated":2166784,"data_stored":1580516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002184"},"pg_stats":[{"pgid":"1.0","version":"22'97","reported_seq":137,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-03-08T00:37:37.292814+0000","last_change":"2022-03-08T00:37:33.240322+0000","last_active":"2022-03-08T00:37:37.292814+0000","last_peered":"2022-03-08T00:37:37.292814+0000","last_clean":"2022-03-08T00:37:37.292814+0000","last_became_active":"2022-03-08T00:37:33.239824+0000","last_became_peered":"2022-03-08T00:37:33.239824+0000","last_unstale":"2022-03-08T00:37:37.292814+0000","last_undegraded":"2022-03-08T00:37:37.292814+0000","last_fullsized":"2022-03-08T00:37:37.292814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T00:37:31.276578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T00:37:31.276578+0000","last_clean_scrub_stamp":"2022-03-08T00:37:31.276578+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T03:58:23.942270+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6220,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738932,"statfs":{"total":95995035648,"available":95988666368,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]}]},{"osd":1,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5964,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739188,"statfs":{"total":95995035648,"available":95988928512,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5900,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739252,"statfs":{"total":95995035648,"available":95988994048,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4992,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740160,"statfs":{"total":95995035648,"available":95989923840,"internally_reserved":0,"allocated":188416,"data_stored":44912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T00:37:45.265 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph pg dump --format=json 2022-03-08T00:37:45.770 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:45.773 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2022-03-08T00:37:45.789 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-08T00:37:44.199514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23076,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":374957532,"statfs":{"total":383980142592,"available":383956512768,"internally_reserved":0,"allocated":2166784,"data_stored":1580516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002184"},"pg_stats":[{"pgid":"1.0","version":"22'97","reported_seq":137,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-03-08T00:37:37.292814+0000","last_change":"2022-03-08T00:37:33.240322+0000","last_active":"2022-03-08T00:37:37.292814+0000","last_peered":"2022-03-08T00:37:37.292814+0000","last_clean":"2022-03-08T00:37:37.292814+0000","last_became_active":"2022-03-08T00:37:33.239824+0000","last_became_peered":"2022-03-08T00:37:33.239824+0000","last_unstale":"2022-03-08T00:37:37.292814+0000","last_undegraded":"2022-03-08T00:37:37.292814+0000","last_fullsized":"2022-03-08T00:37:37.292814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T00:37:31.276578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T00:37:31.276578+0000","last_clean_scrub_stamp":"2022-03-08T00:37:31.276578+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T03:58:23.942270+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6220,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738932,"statfs":{"total":95995035648,"available":95988666368,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]}]},{"osd":1,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5964,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739188,"statfs":{"total":95995035648,"available":95988928512,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5900,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739252,"statfs":{"total":95995035648,"available":95988994048,"internally_reserved":0,"allocated":659456,"data_stored":511868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4992,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740160,"statfs":{"total":95995035648,"available":95989923840,"internally_reserved":0,"allocated":188416,"data_stored":44912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T00:37:45.790 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T00:37:45.790 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T00:37:45.790 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T00:37:45.791 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph health --format=json 2022-03-08T00:37:46.449 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:46.449 INFO:teuthology.orchestra.run.smithi120.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T00:37:46.473 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T00:37:46.473 INFO:tasks.rook:Rook complete, yielding 2022-03-08T00:37:46.474 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T00:37:46.486 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph -s 2022-03-08T00:37:47.114 INFO:teuthology.orchestra.run.smithi120.stdout: cluster: 2022-03-08T00:37:47.114 INFO:teuthology.orchestra.run.smithi120.stdout: id: 034a23aa-8c1d-44a2-ad04-5ada309e2217 2022-03-08T00:37:47.114 INFO:teuthology.orchestra.run.smithi120.stdout: health: HEALTH_OK 2022-03-08T00:37:47.115 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:47.115 INFO:teuthology.orchestra.run.smithi120.stdout: services: 2022-03-08T00:37:47.115 INFO:teuthology.orchestra.run.smithi120.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-08T00:37:47.115 INFO:teuthology.orchestra.run.smithi120.stdout: mgr: a(active, since 2m) 2022-03-08T00:37:47.116 INFO:teuthology.orchestra.run.smithi120.stdout: osd: 4 osds: 4 up (since 18s), 4 in (since 66s) 2022-03-08T00:37:47.116 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:47.116 INFO:teuthology.orchestra.run.smithi120.stdout: data: 2022-03-08T00:37:47.116 INFO:teuthology.orchestra.run.smithi120.stdout: pools: 1 pools, 1 pgs 2022-03-08T00:37:47.116 INFO:teuthology.orchestra.run.smithi120.stdout: objects: 2 objects, 577 KiB 2022-03-08T00:37:47.117 INFO:teuthology.orchestra.run.smithi120.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-03-08T00:37:47.117 INFO:teuthology.orchestra.run.smithi120.stdout: pgs: 1 active+clean 2022-03-08T00:37:47.117 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:47.117 INFO:teuthology.orchestra.run.smithi120.stdout: io: 2022-03-08T00:37:47.118 INFO:teuthology.orchestra.run.smithi120.stdout: client: 921 B/s rd, 26 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-03-08T00:37:47.118 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:37:47.132 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status 2022-03-08T00:37:47.701 INFO:teuthology.orchestra.run.smithi120.stdout:Backend: rook 2022-03-08T00:37:47.702 INFO:teuthology.orchestra.run.smithi120.stdout:Available: Yes 2022-03-08T00:37:47.725 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ps 2022-03-08T00:37:48.267 INFO:teuthology.orchestra.run.smithi120.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T00:37:48.268 INFO:teuthology.orchestra.run.smithi120.stdout:crashcollector.crash smithi120 running (65s) 0s ago 65s - - 4b871cde5e1a 2022-03-08T00:37:48.268 INFO:teuthology.orchestra.run.smithi120.stdout:mgr.a smithi120 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T00:37:48.268 INFO:teuthology.orchestra.run.smithi120.stdout:mon.a smithi120 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T00:37:48.269 INFO:teuthology.orchestra.run.smithi120.stdout:osd.0 smithi120 running (65s) 0s ago 66s - - 4b871cde5e1a 2022-03-08T00:37:48.269 INFO:teuthology.orchestra.run.smithi120.stdout:osd.1 smithi120 running (64s) 0s ago 64s - - 4b871cde5e1a 2022-03-08T00:37:48.269 INFO:teuthology.orchestra.run.smithi120.stdout:osd.2 smithi120 running (61s) 0s ago 62s - - 4b871cde5e1a 2022-03-08T00:37:48.269 INFO:teuthology.orchestra.run.smithi120.stdout:osd.3 smithi120 running (60s) 0s ago 60s - - 4b871cde5e1a 2022-03-08T00:37:48.285 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls 2022-03-08T00:37:48.900 INFO:teuthology.orchestra.run.smithi120.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T00:37:48.900 INFO:teuthology.orchestra.run.smithi120.stdout:crash 1/1 0s ago 65s * 2022-03-08T00:37:48.900 INFO:teuthology.orchestra.run.smithi120.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T00:37:48.900 INFO:teuthology.orchestra.run.smithi120.stdout:mon 1/1 0s ago 3m count:1 2022-03-08T00:37:48.901 INFO:teuthology.orchestra.run.smithi120.stdout:osd 8 0s ago 66s 2022-03-08T00:37:48.901 INFO:teuthology.orchestra.run.smithi120.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T00:37:48.923 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch device ls 2022-03-08T00:37:49.457 INFO:teuthology.orchestra.run.smithi120.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T00:37:49.457 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T00:37:49.457 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T00:37:49.458 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T00:37:49.458 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T00:37:49.477 INFO:teuthology.run_tasks:Running task install... 2022-03-08T00:37:49.490 DEBUG:teuthology.task.install:project ceph 2022-03-08T00:37:49.491 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'}} 2022-03-08T00:37:49.491 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'} 2022-03-08T00:37:49.492 INFO:teuthology.task.install:Using flavor: default 2022-03-08T00:37:49.500 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-08T00:37:49.500 INFO:teuthology.task.install:extra packages: [] 2022-03-08T00:37:49.501 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-03-08T00:37:49.548 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-03-08T00:37:49.553 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T00:37:49.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-key list | grep Ceph 2022-03-08T00:37:49.644 INFO:teuthology.orchestra.run.smithi120.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-08T00:37:49.707 INFO:teuthology.orchestra.run.smithi120.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-08T00:37:49.708 INFO:teuthology.orchestra.run.smithi120.stdout:uid [ unknown] Ceph.com (release key) 2022-03-08T00:37:49.709 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-03-08T00:37:49.758 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-03-08T00:37:49.763 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T00:37:49.764 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-08T00:37:49.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T00:37:49.933 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default/ 2022-03-08T00:37:49.934 INFO:teuthology.task.install.deb:Package version is 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T00:37:50.051 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:37:50.052 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-08T00:37:50.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo apt-get update 2022-03-08T00:37:50.219 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T00:37:50.307 INFO:teuthology.orchestra.run.smithi120.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal InRelease 2022-03-08T00:37:50.329 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-08T00:37:50.333 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T00:37:50.353 INFO:teuthology.orchestra.run.smithi120.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release.gpg 2022-03-08T00:37:50.376 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2022-03-08T00:37:50.409 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main i386 Packages [3,360 B] 2022-03-08T00:37:50.415 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T00:37:50.422 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-03-08T00:37:50.498 INFO:teuthology.orchestra.run.smithi120.stdout:Hit:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T00:37:50.918 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 28.3 kB in 1s (42.9 kB/s) 2022-03-08T00:37:51.869 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:37:52.011 DEBUG:teuthology.orchestra.run.smithi120:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11025-g7a44e9eb-1focal cephadm=17.0.0-11025-g7a44e9eb-1focal ceph-mds=17.0.0-11025-g7a44e9eb-1focal ceph-mgr=17.0.0-11025-g7a44e9eb-1focal ceph-common=17.0.0-11025-g7a44e9eb-1focal ceph-fuse=17.0.0-11025-g7a44e9eb-1focal ceph-test=17.0.0-11025-g7a44e9eb-1focal ceph-volume=17.0.0-11025-g7a44e9eb-1focal radosgw=17.0.0-11025-g7a44e9eb-1focal python3-rados=17.0.0-11025-g7a44e9eb-1focal python3-rgw=17.0.0-11025-g7a44e9eb-1focal python3-cephfs=17.0.0-11025-g7a44e9eb-1focal python3-rbd=17.0.0-11025-g7a44e9eb-1focal libcephfs2=17.0.0-11025-g7a44e9eb-1focal libcephfs-dev=17.0.0-11025-g7a44e9eb-1focal librados2=17.0.0-11025-g7a44e9eb-1focal librbd1=17.0.0-11025-g7a44e9eb-1focal rbd-fuse=17.0.0-11025-g7a44e9eb-1focal 2022-03-08T00:37:52.085 INFO:teuthology.orchestra.run.smithi120.stdout:Reading package lists... 2022-03-08T00:37:52.271 INFO:teuthology.orchestra.run.smithi120.stdout:Building dependency tree... 2022-03-08T00:37:52.272 INFO:teuthology.orchestra.run.smithi120.stdout:Reading state information... 2022-03-08T00:37:52.445 INFO:teuthology.orchestra.run.smithi120.stdout:The following packages were automatically installed and are no longer required: 2022-03-08T00:37:52.446 INFO:teuthology.orchestra.run.smithi120.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-08T00:37:52.446 INFO:teuthology.orchestra.run.smithi120.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-08T00:37:52.447 INFO:teuthology.orchestra.run.smithi120.stdout:The following additional packages will be installed: 2022-03-08T00:37:52.447 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-08T00:37:52.447 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-08T00:37:52.448 INFO:teuthology.orchestra.run.smithi120.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T00:37:52.448 INFO:teuthology.orchestra.run.smithi120.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T00:37:52.448 INFO:teuthology.orchestra.run.smithi120.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T00:37:52.448 INFO:teuthology.orchestra.run.smithi120.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T00:37:52.449 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-08T00:37:52.449 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T00:37:52.449 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T00:37:52.449 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T00:37:52.450 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T00:37:52.450 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T00:37:52.450 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T00:37:52.451 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-08T00:37:52.452 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-08T00:37:52.452 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-08T00:37:52.452 INFO:teuthology.orchestra.run.smithi120.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-08T00:37:52.452 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-08T00:37:52.453 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-08T00:37:52.453 INFO:teuthology.orchestra.run.smithi120.stdout:Suggested packages: 2022-03-08T00:37:52.453 INFO:teuthology.orchestra.run.smithi120.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-08T00:37:52.454 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-08T00:37:52.454 INFO:teuthology.orchestra.run.smithi120.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-08T00:37:52.454 INFO:teuthology.orchestra.run.smithi120.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-08T00:37:52.454 INFO:teuthology.orchestra.run.smithi120.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-08T00:37:52.455 INFO:teuthology.orchestra.run.smithi120.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-08T00:37:52.455 INFO:teuthology.orchestra.run.smithi120.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-08T00:37:52.455 INFO:teuthology.orchestra.run.smithi120.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-08T00:37:52.455 INFO:teuthology.orchestra.run.smithi120.stdout:Recommended packages: 2022-03-08T00:37:52.456 INFO:teuthology.orchestra.run.smithi120.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-08T00:37:52.560 INFO:teuthology.orchestra.run.smithi120.stdout:The following NEW packages will be installed: 2022-03-08T00:37:52.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-08T00:37:52.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-08T00:37:52.562 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-08T00:37:52.563 INFO:teuthology.orchestra.run.smithi120.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T00:37:52.563 INFO:teuthology.orchestra.run.smithi120.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T00:37:52.563 INFO:teuthology.orchestra.run.smithi120.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T00:37:52.563 INFO:teuthology.orchestra.run.smithi120.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T00:37:52.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-08T00:37:52.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T00:37:52.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T00:37:52.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T00:37:52.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T00:37:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T00:37:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T00:37:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-08T00:37:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-08T00:37:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-08T00:37:52.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-08T00:37:52.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-08T00:37:52.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-08T00:37:52.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-08T00:37:52.567 INFO:teuthology.orchestra.run.smithi120.stdout:The following packages will be upgraded: 2022-03-08T00:37:52.568 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 librbd1 2022-03-08T00:37:52.721 INFO:teuthology.orchestra.run.smithi120.stdout:2 upgraded, 95 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T00:37:52.722 INFO:teuthology.orchestra.run.smithi120.stdout:Need to get 181 MB of archives. 2022-03-08T00:37:52.723 INFO:teuthology.orchestra.run.smithi120.stdout:After this operation, 845 MB of additional disk space will be used. 2022-03-08T00:37:52.723 INFO:teuthology.orchestra.run.smithi120.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11025-g7a44e9eb-1focal [3,484 kB] 2022-03-08T00:37:52.756 INFO:teuthology.orchestra.run.smithi120.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-08T00:37:53.021 INFO:teuthology.orchestra.run.smithi120.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11025-g7a44e9eb-1focal [3,533 kB] 2022-03-08T00:37:53.142 INFO:teuthology.orchestra.run.smithi120.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-08T00:37:53.143 INFO:teuthology.orchestra.run.smithi120.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-08T00:37:53.166 INFO:teuthology.orchestra.run.smithi120.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11025-g7a44e9eb-1focal [755 kB] 2022-03-08T00:37:53.174 INFO:teuthology.orchestra.run.smithi120.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-08T00:37:53.204 INFO:teuthology.orchestra.run.smithi120.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11025-g7a44e9eb-1focal [412 kB] 2022-03-08T00:37:53.211 INFO:teuthology.orchestra.run.smithi120.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11025-g7a44e9eb-1focal [22.4 kB] 2022-03-08T00:37:53.211 INFO:teuthology.orchestra.run.smithi120.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11025-g7a44e9eb-1focal [200 kB] 2022-03-08T00:37:53.224 INFO:teuthology.orchestra.run.smithi120.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-08T00:37:53.227 INFO:teuthology.orchestra.run.smithi120.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11025-g7a44e9eb-1focal [41.9 kB] 2022-03-08T00:37:53.229 INFO:teuthology.orchestra.run.smithi120.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11025-g7a44e9eb-1focal [385 kB] 2022-03-08T00:37:53.234 INFO:teuthology.orchestra.run.smithi120.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11025-g7a44e9eb-1focal [4,162 kB] 2022-03-08T00:37:53.321 INFO:teuthology.orchestra.run.smithi120.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-08T00:37:53.327 INFO:teuthology.orchestra.run.smithi120.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-08T00:37:53.346 INFO:teuthology.orchestra.run.smithi120.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-08T00:37:53.350 INFO:teuthology.orchestra.run.smithi120.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-08T00:37:53.365 INFO:teuthology.orchestra.run.smithi120.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-08T00:37:53.368 INFO:teuthology.orchestra.run.smithi120.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-08T00:37:53.369 INFO:teuthology.orchestra.run.smithi120.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-08T00:37:53.371 INFO:teuthology.orchestra.run.smithi120.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11025-g7a44e9eb-1focal [116 kB] 2022-03-08T00:37:53.374 INFO:teuthology.orchestra.run.smithi120.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11025-g7a44e9eb-1focal [478 kB] 2022-03-08T00:37:53.387 INFO:teuthology.orchestra.run.smithi120.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-08T00:37:53.388 INFO:teuthology.orchestra.run.smithi120.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-08T00:37:53.389 INFO:teuthology.orchestra.run.smithi120.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11025-g7a44e9eb-1focal [24.4 MB] 2022-03-08T00:37:53.468 INFO:teuthology.orchestra.run.smithi120.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-08T00:37:53.469 INFO:teuthology.orchestra.run.smithi120.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-08T00:37:53.470 INFO:teuthology.orchestra.run.smithi120.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-08T00:37:53.472 INFO:teuthology.orchestra.run.smithi120.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-08T00:37:53.473 INFO:teuthology.orchestra.run.smithi120.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-08T00:37:53.477 INFO:teuthology.orchestra.run.smithi120.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-08T00:37:53.480 INFO:teuthology.orchestra.run.smithi120.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-08T00:37:53.550 INFO:teuthology.orchestra.run.smithi120.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-08T00:37:53.568 INFO:teuthology.orchestra.run.smithi120.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-08T00:37:53.569 INFO:teuthology.orchestra.run.smithi120.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-08T00:37:53.631 INFO:teuthology.orchestra.run.smithi120.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-08T00:37:53.633 INFO:teuthology.orchestra.run.smithi120.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-08T00:37:53.636 INFO:teuthology.orchestra.run.smithi120.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-08T00:37:53.644 INFO:teuthology.orchestra.run.smithi120.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-08T00:37:53.644 INFO:teuthology.orchestra.run.smithi120.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-08T00:37:53.648 INFO:teuthology.orchestra.run.smithi120.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-08T00:37:53.652 INFO:teuthology.orchestra.run.smithi120.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-08T00:37:53.713 INFO:teuthology.orchestra.run.smithi120.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-08T00:37:53.714 INFO:teuthology.orchestra.run.smithi120.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-08T00:37:53.716 INFO:teuthology.orchestra.run.smithi120.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-08T00:37:54.072 INFO:teuthology.orchestra.run.smithi120.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-08T00:37:54.131 INFO:teuthology.orchestra.run.smithi120.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-08T00:37:54.133 INFO:teuthology.orchestra.run.smithi120.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-08T00:37:54.268 INFO:teuthology.orchestra.run.smithi120.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-08T00:37:54.268 INFO:teuthology.orchestra.run.smithi120.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-08T00:37:54.341 INFO:teuthology.orchestra.run.smithi120.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-08T00:37:54.341 INFO:teuthology.orchestra.run.smithi120.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11025-g7a44e9eb-1focal [6,433 kB] 2022-03-08T00:37:54.342 INFO:teuthology.orchestra.run.smithi120.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-08T00:37:54.343 INFO:teuthology.orchestra.run.smithi120.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-08T00:37:54.345 INFO:teuthology.orchestra.run.smithi120.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-08T00:37:54.345 INFO:teuthology.orchestra.run.smithi120.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-08T00:37:54.347 INFO:teuthology.orchestra.run.smithi120.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-08T00:37:54.348 INFO:teuthology.orchestra.run.smithi120.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-08T00:37:54.423 INFO:teuthology.orchestra.run.smithi120.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-08T00:37:54.429 INFO:teuthology.orchestra.run.smithi120.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-08T00:37:54.434 INFO:teuthology.orchestra.run.smithi120.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11025-g7a44e9eb-1focal [211 kB] 2022-03-08T00:37:54.439 INFO:teuthology.orchestra.run.smithi120.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11025-g7a44e9eb-1focal [151 kB] 2022-03-08T00:37:54.445 INFO:teuthology.orchestra.run.smithi120.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11025-g7a44e9eb-1focal [1,574 kB] 2022-03-08T00:37:54.496 INFO:teuthology.orchestra.run.smithi120.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11025-g7a44e9eb-1focal [7,824 kB] 2022-03-08T00:37:54.506 INFO:teuthology.orchestra.run.smithi120.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-08T00:37:54.507 INFO:teuthology.orchestra.run.smithi120.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-08T00:37:54.509 INFO:teuthology.orchestra.run.smithi120.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-08T00:37:54.511 INFO:teuthology.orchestra.run.smithi120.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-08T00:37:54.514 INFO:teuthology.orchestra.run.smithi120.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-08T00:37:54.520 INFO:teuthology.orchestra.run.smithi120.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-08T00:37:54.520 INFO:teuthology.orchestra.run.smithi120.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-08T00:37:54.521 INFO:teuthology.orchestra.run.smithi120.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-08T00:37:54.586 INFO:teuthology.orchestra.run.smithi120.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-08T00:37:54.588 INFO:teuthology.orchestra.run.smithi120.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-08T00:37:54.668 INFO:teuthology.orchestra.run.smithi120.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-08T00:37:54.669 INFO:teuthology.orchestra.run.smithi120.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-08T00:37:54.671 INFO:teuthology.orchestra.run.smithi120.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-08T00:37:54.671 INFO:teuthology.orchestra.run.smithi120.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-08T00:37:54.672 INFO:teuthology.orchestra.run.smithi120.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-08T00:37:54.673 INFO:teuthology.orchestra.run.smithi120.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-08T00:37:54.673 INFO:teuthology.orchestra.run.smithi120.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-08T00:37:54.674 INFO:teuthology.orchestra.run.smithi120.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-08T00:37:54.749 INFO:teuthology.orchestra.run.smithi120.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-08T00:37:54.761 INFO:teuthology.orchestra.run.smithi120.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11025-g7a44e9eb-1focal [28.6 MB] 2022-03-08T00:37:55.838 INFO:teuthology.orchestra.run.smithi120.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11025-g7a44e9eb-1focal [3,808 B] 2022-03-08T00:37:55.838 INFO:teuthology.orchestra.run.smithi120.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [863 kB] 2022-03-08T00:37:55.843 INFO:teuthology.orchestra.run.smithi120.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11025-g7a44e9eb-1focal [2,396 kB] 2022-03-08T00:37:55.861 INFO:teuthology.orchestra.run.smithi120.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11025-g7a44e9eb-1focal [1,677 kB] 2022-03-08T00:37:55.883 INFO:teuthology.orchestra.run.smithi120.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11025-g7a44e9eb-1focal [7,889 kB] 2022-03-08T00:37:56.143 INFO:teuthology.orchestra.run.smithi120.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11025-g7a44e9eb-1focal [3,924 B] 2022-03-08T00:37:56.143 INFO:teuthology.orchestra.run.smithi120.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11025-g7a44e9eb-1focal [50.5 MB] 2022-03-08T00:37:58.059 INFO:teuthology.orchestra.run.smithi120.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11025-g7a44e9eb-1focal [219 kB] 2022-03-08T00:37:58.134 INFO:teuthology.orchestra.run.smithi120.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11025-g7a44e9eb-1focal [69.5 kB] 2022-03-08T00:37:58.134 INFO:teuthology.orchestra.run.smithi120.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11025-g7a44e9eb-1focal [21.7 kB] 2022-03-08T00:37:58.135 INFO:teuthology.orchestra.run.smithi120.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11025-g7a44e9eb-1focal [12.7 MB] 2022-03-08T00:37:58.433 INFO:teuthology.orchestra.run.smithi120.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [78.8 kB] 2022-03-08T00:37:59.121 INFO:teuthology.orchestra.run.smithi120.stdout:Fetched 181 MB in 6s (30.5 MB/s) 2022-03-08T00:37:59.300 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-08T00:37:59.346 INFO:teuthology.orchestra.run.smithi120.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-08T00:37:59.349 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-08T00:37:59.434 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T00:37:59.877 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../01-librbd1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:00.052 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librbd1 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T00:38:00.812 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../02-librados2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:00.942 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librados2 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T00:38:01.670 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libcephfs2. 2022-03-08T00:38:01.682 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:01.753 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:02.109 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rados. 2022-03-08T00:38:02.122 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:02.184 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:02.734 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-08T00:38:02.748 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:02.817 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:03.064 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cephfs. 2022-03-08T00:38:03.076 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:03.114 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:03.483 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-08T00:38:03.494 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:03.558 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:03.877 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-prettytable. 2022-03-08T00:38:03.890 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-08T00:38:03.930 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-08T00:38:04.188 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rbd. 2022-03-08T00:38:04.201 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:04.231 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:04.786 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-08T00:38:04.800 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-08T00:38:04.853 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T00:38:05.298 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-08T00:38:05.312 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-08T00:38:05.357 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T00:38:05.692 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-08T00:38:05.705 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-08T00:38:05.741 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T00:38:06.119 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package librgw2. 2022-03-08T00:38:06.129 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../13-librgw2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:06.170 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:06.896 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rgw. 2022-03-08T00:38:06.911 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:06.946 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:07.348 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liboath0. 2022-03-08T00:38:07.359 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-08T00:38:07.419 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-08T00:38:07.776 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libradosstriper1. 2022-03-08T00:38:07.789 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:07.829 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:08.429 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-common. 2022-03-08T00:38:08.438 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:08.548 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:11.375 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-base. 2022-03-08T00:38:11.388 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:11.448 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:12.734 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cryptography. 2022-03-08T00:38:12.747 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-08T00:38:12.797 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T00:38:13.193 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-openssl. 2022-03-08T00:38:13.206 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-08T00:38:13.244 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-08T00:38:13.567 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-08T00:38:13.579 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:13.617 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:14.029 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-08T00:38:14.042 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-08T00:38:14.079 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T00:38:14.513 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-08T00:38:14.526 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-08T00:38:14.603 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-08T00:38:15.091 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-jwt. 2022-03-08T00:38:15.099 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-08T00:38:15.166 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T00:38:15.453 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-logutils. 2022-03-08T00:38:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-08T00:38:15.503 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-08T00:38:15.857 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-08T00:38:15.871 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-08T00:38:15.939 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-08T00:38:16.409 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-mako. 2022-03-08T00:38:16.420 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-08T00:38:16.460 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T00:38:16.940 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-08T00:38:16.954 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-08T00:38:16.986 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-08T00:38:17.460 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-08T00:38:17.468 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-08T00:38:17.518 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T00:38:17.873 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webob. 2022-03-08T00:38:17.885 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-08T00:38:17.927 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-08T00:38:18.281 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-08T00:38:18.289 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-08T00:38:18.322 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T00:38:18.774 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-bs4. 2022-03-08T00:38:18.788 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-08T00:38:18.841 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-08T00:38:19.229 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-waitress. 2022-03-08T00:38:19.244 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-08T00:38:19.306 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-08T00:38:19.700 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-tempita. 2022-03-08T00:38:19.713 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-08T00:38:19.754 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-08T00:38:20.132 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-paste. 2022-03-08T00:38:20.367 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-08T00:38:20.367 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T00:38:20.549 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-08T00:38:20.563 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-08T00:38:20.599 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T00:38:20.950 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-08T00:38:20.963 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-08T00:38:21.000 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-08T00:38:21.462 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webtest. 2022-03-08T00:38:21.474 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-08T00:38:21.521 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-08T00:38:21.899 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pecan. 2022-03-08T00:38:21.913 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-08T00:38:21.957 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T00:38:22.363 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-08T00:38:22.376 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-08T00:38:22.473 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T00:38:22.934 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-08T00:38:22.942 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:22.984 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:23.320 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr. 2022-03-08T00:38:23.334 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:23.379 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:25.791 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mon. 2022-03-08T00:38:25.805 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:25.846 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:27.145 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-osd. 2022-03-08T00:38:27.155 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:27.203 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:30.294 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph. 2022-03-08T00:38:30.308 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../45-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:30.377 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:30.879 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-fuse. 2022-03-08T00:38:30.887 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:30.928 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:31.294 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mds. 2022-03-08T00:38:31.307 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:31.364 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:32.042 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-08T00:38:32.052 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-08T00:38:32.117 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-08T00:38:32.628 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjs-underscore. 2022-03-08T00:38:32.641 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-08T00:38:32.686 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T00:38:32.972 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-08T00:38:32.984 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-08T00:38:33.022 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T00:38:33.331 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-routes. 2022-03-08T00:38:33.340 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-08T00:38:33.381 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-08T00:38:33.862 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-08T00:38:33.874 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:33.929 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:34.482 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-decorator. 2022-03-08T00:38:34.495 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-08T00:38:34.523 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T00:38:34.976 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-08T00:38:34.992 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-08T00:38:35.034 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T00:38:35.469 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-scipy. 2022-03-08T00:38:35.482 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-08T00:38:35.527 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-08T00:38:37.444 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-08T00:38:37.458 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-08T00:38:37.502 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T00:38:38.159 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-joblib. 2022-03-08T00:38:38.173 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-08T00:38:38.217 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-08T00:38:38.765 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-sklearn. 2022-03-08T00:38:38.778 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-08T00:38:38.814 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T00:38:39.655 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-08T00:38:39.669 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:39.713 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:41.128 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-cachetools. 2022-03-08T00:38:41.141 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-08T00:38:41.191 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-08T00:38:41.511 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-08T00:38:41.524 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-08T00:38:41.569 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T00:38:41.974 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-08T00:38:41.987 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-08T00:38:42.023 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T00:38:42.548 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-rsa. 2022-03-08T00:38:42.561 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-08T00:38:42.626 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-08T00:38:42.972 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-google-auth. 2022-03-08T00:38:42.985 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-08T00:38:43.021 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-08T00:38:43.398 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-blinker. 2022-03-08T00:38:43.407 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-08T00:38:43.465 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T00:38:43.793 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-08T00:38:43.806 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-08T00:38:43.851 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T00:38:44.214 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-08T00:38:44.229 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-08T00:38:44.289 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T00:38:44.823 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-websocket. 2022-03-08T00:38:44.836 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-08T00:38:44.883 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T00:38:45.195 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-08T00:38:45.210 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-08T00:38:45.253 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T00:38:45.950 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-08T00:38:45.966 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:46.008 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:46.370 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-08T00:38:46.382 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-08T00:38:46.420 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-08T00:38:47.009 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-08T00:38:47.024 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T00:38:47.067 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T00:38:47.402 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package jq. 2022-03-08T00:38:47.418 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T00:38:47.467 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T00:38:47.927 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-08T00:38:47.941 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-08T00:38:47.984 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T00:38:48.271 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package xmlstarlet. 2022-03-08T00:38:48.284 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-08T00:38:48.388 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-08T00:38:48.809 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-test. 2022-03-08T00:38:48.821 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:48.901 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:53.801 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package ceph-volume. 2022-03-08T00:38:53.815 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-08T00:38:53.887 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:54.468 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package cephadm. 2022-03-08T00:38:54.477 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../78-cephadm_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:54.554 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:54.831 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-08T00:38:54.844 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:38:54.900 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:38:55.452 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package nvme-cli. 2022-03-08T00:38:55.461 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-08T00:38:55.524 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T00:38:56.100 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-08T00:38:56.114 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-08T00:38:56.288 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T00:38:57.237 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-attr. 2022-03-08T00:38:57.246 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-08T00:38:57.303 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-08T00:38:57.791 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-webencodings. 2022-03-08T00:38:57.804 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-08T00:38:57.874 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T00:38:58.228 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-html5lib. 2022-03-08T00:38:58.241 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-08T00:38:58.270 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-08T00:38:58.825 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-08T00:38:58.836 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-08T00:38:58.874 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T00:38:59.447 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-packaging. 2022-03-08T00:38:59.456 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-08T00:38:59.502 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-08T00:38:59.890 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pastescript. 2022-03-08T00:38:59.904 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-08T00:38:59.966 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-08T00:39:00.352 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pluggy. 2022-03-08T00:39:00.365 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-08T00:39:00.468 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-08T00:39:00.823 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-psutil. 2022-03-08T00:39:00.837 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-08T00:39:00.881 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T00:39:01.275 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-py. 2022-03-08T00:39:01.288 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-08T00:39:01.338 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T00:39:01.995 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-08T00:39:02.011 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-08T00:39:02.099 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T00:39:02.459 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-08T00:39:02.472 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-08T00:39:02.561 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T00:39:02.907 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-pytest. 2022-03-08T00:39:02.916 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-08T00:39:02.962 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-08T00:39:03.503 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package python3-simplejson. 2022-03-08T00:39:03.518 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-08T00:39:03.577 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T00:39:04.048 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package radosgw. 2022-03-08T00:39:04.062 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../95-radosgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:39:04.131 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:05.651 INFO:teuthology.orchestra.run.smithi120.stdout:Selecting previously unselected package rbd-fuse. 2022-03-08T00:39:05.668 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-08T00:39:05.725 INFO:teuthology.orchestra.run.smithi120.stdout:Unpacking rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:06.037 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-08T00:39:06.415 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T00:39:06.541 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T00:39:06.742 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:07.563 INFO:teuthology.orchestra.run.smithi120.stdout:Adding system user cephadm....done 2022-03-08T00:39:07.798 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-08T00:39:08.176 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-08T00:39:08.458 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-08T00:39:08.619 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T00:39:08.962 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-08T00:39:09.352 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T00:39:09.637 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-08T00:39:09.931 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T00:39:10.221 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T00:39:10.607 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:10.923 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T00:39:11.075 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-08T00:39:11.367 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T00:39:11.659 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T00:39:12.061 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T00:39:12.401 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T00:39:12.870 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T00:39:13.171 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-08T00:39:13.462 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-08T00:39:13.763 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T00:39:13.923 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T00:39:14.214 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T00:39:14.631 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T00:39:15.009 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T00:39:15.405 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:15.710 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-08T00:39:16.096 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T00:39:16.263 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-08T00:39:16.546 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T00:39:16.688 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T00:39:16.831 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T00:39:17.290 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T00:39:17.559 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-08T00:39:17.892 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T00:39:18.076 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-08T00:39:18.486 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-08T00:39:18.813 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T00:39:19.138 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T00:39:19.299 INFO:teuthology.orchestra.run.smithi120.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-08T00:39:19.563 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-08T00:39:19.723 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T00:39:20.066 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-08T00:39:20.473 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T00:39:20.735 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T00:39:20.934 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-08T00:39:21.277 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T00:39:21.570 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librados2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:21.702 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-08T00:39:23.382 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:23.525 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:23.649 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T00:39:23.801 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-08T00:39:24.216 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-08T00:39:24.529 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-08T00:39:24.830 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-08T00:39:25.214 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:25.340 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T00:39:25.703 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:25.829 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T00:39:26.240 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T00:39:27.407 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up librbd1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:27.542 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T00:39:28.222 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-08T00:39:28.828 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T00:39:29.172 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T00:39:29.331 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-08T00:39:29.633 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-08T00:39:29.959 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:30.086 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:30.306 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T00:39:30.307 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T00:39:31.277 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:31.464 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T00:39:31.782 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-08T00:39:32.319 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T00:39:32.819 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:32.972 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:33.169 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:33.312 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-08T00:39:33.618 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-08T00:39:33.917 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:34.029 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:34.156 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T00:39:35.230 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:35.866 INFO:teuthology.orchestra.run.smithi120.stdout:Adding group ceph....done 2022-03-08T00:39:36.322 INFO:teuthology.orchestra.run.smithi120.stdout:Adding system user ceph....done 2022-03-08T00:39:36.511 INFO:teuthology.orchestra.run.smithi120.stdout:Setting system user ceph properties....done 2022-03-08T00:39:36.529 INFO:teuthology.orchestra.run.smithi120.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-08T00:39:36.654 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-08T00:39:37.238 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-08T00:39:38.197 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:38.396 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:38.761 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T00:39:38.761 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T00:39:40.166 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:40.569 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-08T00:39:41.554 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:41.804 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T00:39:41.804 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T00:39:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:43.125 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T00:39:43.125 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T00:39:44.134 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:44.957 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T00:39:44.957 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T00:39:45.918 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:46.102 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:46.245 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:46.465 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T00:39:46.465 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T00:39:47.449 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:47.600 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:47.798 INFO:teuthology.orchestra.run.smithi120.stdout:Setting up ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-08T00:39:47.959 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-08T00:39:48.511 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T00:39:51.235 INFO:teuthology.orchestra.run.smithi120.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T00:39:51.696 INFO:teuthology.orchestra.run.smithi120.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-08T00:39:51.699 DEBUG:teuthology.parallel:result is None 2022-03-08T00:39:51.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2022-03-08T00:39:51.744 INFO:teuthology.orchestra.run.smithi120.stdout:Ubuntu 2022-03-08T00:39:51.749 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T00:39:51.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T00:39:51.908 DEBUG:teuthology.orchestra.run.smithi120:> dpkg-query -W -f '${Version}' ceph 2022-03-08T00:39:51.931 INFO:teuthology.orchestra.run.smithi120.stdout:17.0.0-11025-g7a44e9eb-1focal 2022-03-08T00:39:51.932 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T00:39:51.933 INFO:teuthology.task.install:The correct ceph version 17.0.0-11025-g7a44e9eb-1focal is installed. 2022-03-08T00:39:51.933 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-08T00:39:51.934 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:39:51.935 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-08T00:39:51.954 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-08T00:39:51.955 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:39:51.955 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2022-03-08T00:39:51.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-08T00:39:52.031 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-08T00:39:52.032 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2022-03-08T00:39:52.032 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-08T00:39:52.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-08T00:39:52.061 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-08T00:39:52.078 INFO:tasks.radosbench:Beginning radosbench... 2022-03-08T00:39:52.079 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-08T00:39:52.080 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd pool create unique_pool_0 16 2022-03-08T00:39:53.599 INFO:teuthology.orchestra.run.smithi120.stderr:pool 'unique_pool_0' created 2022-03-08T00:39:53.622 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-08T00:39:54.781 INFO:teuthology.orchestra.run.smithi120.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-08T00:39:55.796 DEBUG:teuthology.orchestra.run.smithi120:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-08T00:39:55.799 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T00:39:55.816 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch host label add '`hostname`' 'foo 2022-03-08T00:39:55.817 DEBUG:teuthology.orchestra.run.smithi120:> ceph' orch host ls '|' grep 'foo 2022-03-08T00:39:55.817 DEBUG:teuthology.orchestra.run.smithi120:> ceph' orch host label rm '`hostname`' 'foo 2022-03-08T00:39:55.817 DEBUG:teuthology.orchestra.run.smithi120:> ceph' orch host ls '|' grep -v 'foo 2022-03-08T00:39:55.818 DEBUG:teuthology.orchestra.run.smithi120:> ' 2022-03-08T00:39:55.849 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:hints = 1 2022-03-08T00:39:55.850 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-08T00:39:55.850 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Object prefix: benchmark_data_smithi120_51949 2022-03-08T00:39:55.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:39:55.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 0 0 0 0 0 0 - 0 2022-03-08T00:39:56.289 INFO:teuthology.orchestra.run.smithi120.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-08T00:39:56.311 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T00:39:56.322 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T00:39:56.322 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi120.front.sepia.ceph.com 2022-03-08T00:39:56.323 DEBUG:teuthology.orchestra.run.smithi120:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T00:39:56.323 DEBUG:teuthology.orchestra.run.smithi120:> toolbox() { 2022-03-08T00:39:56.323 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T00:39:56.324 DEBUG:teuthology.orchestra.run.smithi120:> } 2022-03-08T00:39:56.324 DEBUG:teuthology.orchestra.run.smithi120:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T00:39:56.324 DEBUG:teuthology.orchestra.run.smithi120:> toolbox ceph orch osd rm 0 --force 2022-03-08T00:39:56.325 DEBUG:teuthology.orchestra.run.smithi120:> removed_pv="" 2022-03-08T00:39:56.325 DEBUG:teuthology.orchestra.run.smithi120:> while [ "$removed_pv" = "" ] 2022-03-08T00:39:56.325 DEBUG:teuthology.orchestra.run.smithi120:> do 2022-03-08T00:39:56.325 DEBUG:teuthology.orchestra.run.smithi120:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T00:39:56.326 DEBUG:teuthology.orchestra.run.smithi120:> sleep 3s 2022-03-08T00:39:56.326 DEBUG:teuthology.orchestra.run.smithi120:> done 2022-03-08T00:39:56.326 DEBUG:teuthology.orchestra.run.smithi120:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T00:39:56.327 DEBUG:teuthology.orchestra.run.smithi120:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T00:39:56.327 DEBUG:teuthology.orchestra.run.smithi120:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T00:39:56.327 DEBUG:teuthology.orchestra.run.smithi120:> zap_completion="0" 2022-03-08T00:39:56.327 DEBUG:teuthology.orchestra.run.smithi120:> while [ "$zap_completion" = "0" ] 2022-03-08T00:39:56.328 DEBUG:teuthology.orchestra.run.smithi120:> do 2022-03-08T00:39:56.328 DEBUG:teuthology.orchestra.run.smithi120:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T00:39:56.328 DEBUG:teuthology.orchestra.run.smithi120:> sleep 3s 2022-03-08T00:39:56.329 DEBUG:teuthology.orchestra.run.smithi120:> done 2022-03-08T00:39:56.329 DEBUG:teuthology.orchestra.run.smithi120:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T00:39:56.329 DEBUG:teuthology.orchestra.run.smithi120:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T00:39:56.329 DEBUG:teuthology.orchestra.run.smithi120:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T00:39:56.330 DEBUG:teuthology.orchestra.run.smithi120:> num_osd="0" 2022-03-08T00:39:56.330 DEBUG:teuthology.orchestra.run.smithi120:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T00:39:56.330 DEBUG:teuthology.orchestra.run.smithi120:> do 2022-03-08T00:39:56.331 DEBUG:teuthology.orchestra.run.smithi120:> echo "waiting for osd to come back up" 2022-03-08T00:39:56.331 DEBUG:teuthology.orchestra.run.smithi120:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T00:39:56.332 DEBUG:teuthology.orchestra.run.smithi120:> sleep 30s 2022-03-08T00:39:56.332 DEBUG:teuthology.orchestra.run.smithi120:> done 2022-03-08T00:39:56.332 DEBUG:teuthology.orchestra.run.smithi120:> ' 2022-03-08T00:39:56.365 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:39:56.367 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:39:56.367 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:39:56.850 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 1 16 1289 1273 79.5582 79.5625 0.00746437 0.0125011 2022-03-08T00:39:57.850 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 2 16 2565 2549 79.6504 79.75 0.0093227 0.0125029 2022-03-08T00:39:58.850 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 3 16 4047 4031 83.9725 92.625 0.0167904 0.011871 2022-03-08T00:39:59.148 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:39:59.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 4 16 5304 5288 82.6181 78.5625 0.00912741 0.0120855 2022-03-08T00:39:59.970 INFO:teuthology.orchestra.run.smithi120.stderr:+ orig_num_osd=4 2022-03-08T00:39:59.971 INFO:teuthology.orchestra.run.smithi120.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T00:39:59.971 INFO:teuthology.orchestra.run.smithi120.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T00:40:00.243 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:40:00.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 5 16 6377 6361 79.5057 67.0625 0.0073854 0.0125544 2022-03-08T00:40:01.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 6 16 7450 7434 77.4307 67.0625 0.0113907 0.0128924 2022-03-08T00:40:02.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:02.640546 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T00:40:02.643 INFO:tasks.rook.operator.smithi120.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T00:40:02.643 INFO:tasks.rook.operator.smithi120.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T00:40:02.643 INFO:tasks.rook.operator.smithi120.stdout: ... // 3 identical fields 2022-03-08T00:40:02.643 INFO:tasks.rook.operator.smithi120.stdout: Config: nil, 2022-03-08T00:40:02.644 INFO:tasks.rook.operator.smithi120.stdout: Selection: {}, 2022-03-08T00:40:02.644 INFO:tasks.rook.operator.smithi120.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T00:40:02.644 INFO:tasks.rook.operator.smithi120.stdout:- { 2022-03-08T00:40:02.644 INFO:tasks.rook.operator.smithi120.stdout:- Name: "smithi120-lv-1", 2022-03-08T00:40:02.645 INFO:tasks.rook.operator.smithi120.stdout:- Count: 1, 2022-03-08T00:40:02.645 INFO:tasks.rook.operator.smithi120.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T00:40:02.645 INFO:tasks.rook.operator.smithi120.stdout:- }, 2022-03-08T00:40:02.645 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:02.646 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:02.646 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:02.649 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:40:02.649 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:40:02.650 INFO:tasks.rook.operator.smithi120.stdout: Annotations: nil, 2022-03-08T00:40:02.650 INFO:tasks.rook.operator.smithi120.stdout: Labels: nil, 2022-03-08T00:40:02.650 INFO:tasks.rook.operator.smithi120.stdout: ... // 20 identical fields 2022-03-08T00:40:02.650 INFO:tasks.rook.operator.smithi120.stdout: } 2022-03-08T00:40:02.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:02.640636 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:40:02.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 7 16 8503 8487 75.7701 65.8125 0.00863101 0.0131806 2022-03-08T00:40:03.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:03.694048 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:40:03.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 8 16 9448 9432 73.6809 59.0625 0.0094046 0.0135598 2022-03-08T00:40:04.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 9 16 10351 10335 71.7643 56.4375 0.00369429 0.0138823 2022-03-08T00:40:05.075 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:05.010608 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:40:05.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 10 16 10746 10730 67.0563 24.6875 0.00619392 0.0140607 2022-03-08T00:40:06.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 11 16 10746 10730 60.9602 0 - 0.0140607 2022-03-08T00:40:07.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 12 16 10746 10730 55.8801 0 - 0.0140607 2022-03-08T00:40:08.851 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 13 16 10746 10730 51.5816 0 - 0.0140607 2022-03-08T00:40:09.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 14 16 10746 10730 47.8972 0 - 0.0140607 2022-03-08T00:40:10.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 15 16 10746 10730 44.704 0 - 0.0140607 2022-03-08T00:40:11.124 INFO:teuthology.orchestra.run.smithi120.stdout:Success 2022-03-08T00:40:11.143 INFO:teuthology.orchestra.run.smithi120.stderr:+ removed_pv= 2022-03-08T00:40:11.143 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' '' = '' ']' 2022-03-08T00:40:11.144 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl get pv 2022-03-08T00:40:11.144 INFO:teuthology.orchestra.run.smithi120.stderr:++ grep Released 2022-03-08T00:40:11.145 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f1 -d ' ' 2022-03-08T00:40:11.477 INFO:teuthology.orchestra.run.smithi120.stderr:+ removed_pv= 2022-03-08T00:40:11.477 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 3s 2022-03-08T00:40:11.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 16 16 10746 10730 41.9099 0 - 0.0140607 2022-03-08T00:40:12.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 17 16 10847 10831 39.8159 0.901786 0.00221506 0.0197091 2022-03-08T00:40:13.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 18 16 10847 10831 37.6036 0 - 0.0197091 2022-03-08T00:40:14.478 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' '' = '' ']' 2022-03-08T00:40:14.479 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl get pv 2022-03-08T00:40:14.479 INFO:teuthology.orchestra.run.smithi120.stderr:++ grep Released 2022-03-08T00:40:14.480 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f1 -d ' ' 2022-03-08T00:40:14.691 INFO:teuthology.orchestra.run.smithi120.stderr:+ removed_pv= 2022-03-08T00:40:14.692 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 3s 2022-03-08T00:40:14.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 19 16 11609 11593 38.1308 23.8125 0.0144754 0.0262152 2022-03-08T00:40:15.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:40:15.852828+0000 min lat: 0.0015392 max lat: 8.64344 avg lat: 0.0263627 2022-03-08T00:40:15.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:40:15.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 20 16 12149 12133 37.9116 33.75 0.0210591 0.0263627 2022-03-08T00:40:16.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 21 16 12979 12963 38.5762 51.875 0.0212235 0.0258855 2022-03-08T00:40:17.576 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:17.573714 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T00:40:17.693 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' '' = '' ']' 2022-03-08T00:40:17.693 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl get pv 2022-03-08T00:40:17.694 INFO:teuthology.orchestra.run.smithi120.stderr:++ grep Released 2022-03-08T00:40:17.694 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f1 -d ' ' 2022-03-08T00:40:17.840 INFO:teuthology.orchestra.run.smithi120.stderr:+ removed_pv=smithi120-lv-1 2022-03-08T00:40:17.841 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 3s 2022-03-08T00:40:17.852 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 22 16 13539 13523 38.4135 35 0.0178525 0.0259576 2022-03-08T00:40:18.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 23 16 14398 14382 39.0774 53.6875 0.0307167 0.0255756 2022-03-08T00:40:19.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 24 16 15134 15118 39.3656 46 0.0160488 0.0253901 2022-03-08T00:40:20.842 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' smithi120-lv-1 = '' ']' 2022-03-08T00:40:20.843 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl get pv smithi120-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-08T00:40:20.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 25 16 15747 15731 39.3233 38.3125 0.0363355 0.0254131 2022-03-08T00:40:20.949 INFO:teuthology.orchestra.run.smithi120.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-08T00:40:20.951 INFO:teuthology.orchestra.run.smithi120.stderr:++ echo smithi120-lv-1 2022-03-08T00:40:20.951 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f1 -d - 2022-03-08T00:40:20.952 INFO:teuthology.orchestra.run.smithi120.stderr:+ host=smithi120 2022-03-08T00:40:20.952 INFO:teuthology.orchestra.run.smithi120.stderr:+ toolbox ceph orch device zap smithi120 /dev/vg_nvme/lv_1 --force 2022-03-08T00:40:20.952 INFO:teuthology.orchestra.run.smithi120.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi120 /dev/vg_nvme/lv_1 --force 2022-03-08T00:40:21.056 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:40:21.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 26 16 16521 16505 39.6713 48.375 0.0238444 0.0251785 2022-03-08T00:40:22.134 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/vg_nvme/lv_1 on smithi120 zapped 2022-03-08T00:40:22.174 INFO:teuthology.orchestra.run.smithi120.stderr:+ zap_completion=0 2022-03-08T00:40:22.175 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 0 = 0 ']' 2022-03-08T00:40:22.175 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T00:40:22.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 27 16 17027 17011 39.3732 31.625 0.00833989 0.0253834 2022-03-08T00:40:22.933 INFO:teuthology.orchestra.run.smithi120.stderr:+ zap_completion= 2022-03-08T00:40:22.934 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 3s 2022-03-08T00:40:23.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 28 16 17713 17697 39.4981 42.875 0.0188374 0.0253089 2022-03-08T00:40:24.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 29 16 18543 18527 39.9247 51.875 0.0202794 0.0250362 2022-03-08T00:40:25.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 30 16 19336 19320 40.2458 49.5625 0.013921 0.0248364 2022-03-08T00:40:25.935 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' '' = 0 ']' 2022-03-08T00:40:25.935 INFO:teuthology.orchestra.run.smithi120.stderr:+ kubectl patch pv smithi120-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-08T00:40:26.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 31 16 20083 20067 40.4535 46.6875 0.0109559 0.0247067 2022-03-08T00:40:27.171 INFO:teuthology.orchestra.run.smithi120.stdout:persistentvolume/smithi120-lv-1 patched 2022-03-08T00:40:27.173 INFO:teuthology.orchestra.run.smithi120.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T00:40:27.173 INFO:teuthology.orchestra.run.smithi120.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T00:40:27.683 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:40:27.853 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 32 16 20781 20765 40.5525 43.625 0.018767 0.0246441 2022-03-08T00:40:28.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 33 16 20938 20922 39.6209 9.8125 0.00219455 0.0245478 2022-03-08T00:40:29.853 INFO:teuthology.orchestra.run.smithi120.stdout:['Success'] 2022-03-08T00:40:29.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 34 16 20938 20922 38.4556 0 - 0.0245478 2022-03-08T00:40:29.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:29.856080 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T00:40:29.857 INFO:tasks.rook.operator.smithi120.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T00:40:29.858 INFO:tasks.rook.operator.smithi120.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T00:40:29.858 INFO:tasks.rook.operator.smithi120.stdout: ... // 3 identical fields 2022-03-08T00:40:29.858 INFO:tasks.rook.operator.smithi120.stdout: Config: nil, 2022-03-08T00:40:29.858 INFO:tasks.rook.operator.smithi120.stdout: Selection: {}, 2022-03-08T00:40:29.859 INFO:tasks.rook.operator.smithi120.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T00:40:29.859 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:29.859 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:29.859 INFO:tasks.rook.operator.smithi120.stdout: {Name: "smithi120-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi120-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T00:40:29.860 INFO:tasks.rook.operator.smithi120.stdout:+ { 2022-03-08T00:40:29.860 INFO:tasks.rook.operator.smithi120.stdout:+ Name: "smithi120-lv-1", 2022-03-08T00:40:29.860 INFO:tasks.rook.operator.smithi120.stdout:+ Count: 1, 2022-03-08T00:40:29.861 INFO:tasks.rook.operator.smithi120.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T00:40:29.861 INFO:tasks.rook.operator.smithi120.stdout:+ }, 2022-03-08T00:40:29.861 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:40:29.861 INFO:tasks.rook.operator.smithi120.stdout: }, 2022-03-08T00:40:29.862 INFO:tasks.rook.operator.smithi120.stdout: Annotations: nil, 2022-03-08T00:40:29.862 INFO:tasks.rook.operator.smithi120.stdout: Labels: nil, 2022-03-08T00:40:29.862 INFO:tasks.rook.operator.smithi120.stdout: ... // 20 identical fields 2022-03-08T00:40:29.862 INFO:tasks.rook.operator.smithi120.stdout: } 2022-03-08T00:40:29.871 INFO:teuthology.orchestra.run.smithi120.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T00:40:30.212 INFO:teuthology.orchestra.run.smithi120.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T00:40:30.419 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:40:30.420 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=0 2022-03-08T00:40:30.421 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 0 '!=' 4 ']' 2022-03-08T00:40:30.421 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:40:30.421 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:40:30.422 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:40:30.422 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:40:30.659 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:40:30.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 35 16 20938 20922 37.3569 0 - 0.0245478 2022-03-08T00:40:31.192 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:40:31.193 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:40:31.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 36 16 20938 20922 36.3192 0 - 0.0245478 2022-03-08T00:40:31.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.919945 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:40:31.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.919972 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:40:31.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.919984 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:40:31.930 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.924824 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:40:31.931 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.926729 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:40:31.931 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:31.926913 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:40:32.477 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:32.476560 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:40:32.773 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:32.773173 I | op-mon: start running mons 2022-03-08T00:40:32.835 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:32.834676 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:40:32.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 37 16 20938 20922 35.3376 0 - 0.0245478 2022-03-08T00:40:32.986 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:32.986220 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:40:33.019 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:33.018732 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:40:33.075 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:33.072860 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:40:33.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:33.602322 I | op-mon: targeting the mon count 1 2022-03-08T00:40:33.674 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:33.620315 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T00:40:33.674 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:33.620391 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:40:33.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 38 16 20938 20922 34.4077 0 - 0.0245478 2022-03-08T00:40:34.185 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:34.182480 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:40:34.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:34.198108 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:40:34.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 39 16 20938 20922 33.5254 0 - 0.0245478 2022-03-08T00:40:35.040 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:35.040299 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T00:40:35.384 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:35.383989 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T00:40:35.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:40:35.854757+0000 min lat: 0.0015392 max lat: 8.64344 avg lat: 0.0245478 2022-03-08T00:40:35.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:40:35.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 40 16 20938 20922 32.6873 0 - 0.0245478 2022-03-08T00:40:36.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 41 16 21256 21240 32.3747 2.48438 0.0213359 0.0308737 2022-03-08T00:40:37.490 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:37.486374 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T00:40:37.854 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 42 16 21917 21901 32.5875 41.3125 0.0262143 0.0306717 2022-03-08T00:40:38.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 43 16 22710 22694 32.9821 49.5625 0.0202769 0.0303035 2022-03-08T00:40:39.502 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:40:39.496631 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T00:40:39.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 44 16 23391 23375 33.1997 42.5625 0.0219685 0.0301036 2022-03-08T00:40:40.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 45 16 24158 24142 33.5271 47.9375 0.0304289 0.0298142 2022-03-08T00:40:41.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 46 16 24765 24749 33.6229 37.9375 0.0114533 0.0297308 2022-03-08T00:40:42.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 47 16 25165 25149 33.4394 25 0.0144995 0.0295726 2022-03-08T00:40:43.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 48 16 25165 25149 32.7428 0 - 0.0295726 2022-03-08T00:40:44.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 49 16 25664 25648 32.711 15.5938 0.0207958 0.0304955 2022-03-08T00:40:45.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 50 16 25664 25648 32.0568 0 - 0.0304955 2022-03-08T00:40:46.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 51 16 26229 26213 32.1205 17.6562 0.0297674 0.0311222 2022-03-08T00:40:47.855 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 52 16 26928 26912 32.3429 43.6875 0.0232509 0.030907 2022-03-08T00:40:48.856 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 53 16 27586 27570 32.508 41.125 0.0178033 0.0307489 2022-03-08T00:40:49.857 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 54 16 28170 28154 32.5819 36.5 0.0244118 0.0306729 2022-03-08T00:40:50.856 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 55 16 28314 28298 32.1531 9 0.0150445 0.0306406 2022-03-08T00:40:51.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 56 16 28636 28620 31.9375 20.125 0.0172834 0.0313 2022-03-08T00:40:52.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 57 16 29253 29237 32.0537 38.5625 0.0200576 0.0311864 2022-03-08T00:40:53.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 58 16 29742 29726 32.0279 30.5625 0.0322441 0.0310661 2022-03-08T00:40:54.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 59 16 30015 29999 31.7743 17.0625 0.0118661 0.0314265 2022-03-08T00:40:55.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:40:55.858825+0000 min lat: 0.0015392 max lat: 8.64344 avg lat: 0.0314157 2022-03-08T00:40:55.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:40:55.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 60 16 30568 30552 31.8207 34.5625 0.017956 0.0314157 2022-03-08T00:40:56.858 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 61 16 31049 31033 31.7918 30.0625 0.026055 0.0314424 2022-03-08T00:40:57.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 62 16 31497 31481 31.7306 28 0.0305346 0.0314681 2022-03-08T00:40:58.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 63 16 31976 31960 31.7021 29.9375 0.0249291 0.0314742 2022-03-08T00:40:59.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 64 16 32620 32604 31.8356 40.25 0.0245913 0.0314003 2022-03-08T00:41:00.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 65 16 33206 33190 31.9092 36.625 0.0216845 0.0313267 2022-03-08T00:41:01.193 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:41:01.194 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:41:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:41:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:41:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:41:01.195 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:41:01.246 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:41:01.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 66 16 33773 33757 31.9626 35.4375 0.0176837 0.0312766 2022-03-08T00:41:02.140 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:41:02.141 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:41:02.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 67 16 34283 34267 31.9613 31.875 0.0247779 0.0312066 2022-03-08T00:41:03.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 68 16 34283 34267 31.4913 0 - 0.0312066 2022-03-08T00:41:04.859 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 69 16 34665 34649 31.3809 11.9375 0.0321438 0.0318574 2022-03-08T00:41:05.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 70 16 35222 35206 31.4292 34.8125 0.0249173 0.0317845 2022-03-08T00:41:06.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 71 16 35814 35798 31.5076 37 0.0351928 0.0317291 2022-03-08T00:41:07.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 72 16 36403 36387 31.5813 36.8125 0.0247622 0.0316571 2022-03-08T00:41:08.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 73 16 36929 36913 31.5989 32.875 0.0228961 0.0316377 2022-03-08T00:41:09.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 74 16 37520 37504 31.671 36.9375 0.0242678 0.0315653 2022-03-08T00:41:10.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 75 16 38052 38036 31.692 33.25 0.0307234 0.0315411 2022-03-08T00:41:11.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 76 16 38729 38713 31.8317 42.3125 0.0139526 0.031406 2022-03-08T00:41:12.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 77 16 38869 38853 31.532 8.75 0.0161703 0.0313726 2022-03-08T00:41:13.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 78 16 38869 38853 31.1277 0 - 0.0313726 2022-03-08T00:41:14.861 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 79 16 39386 39370 31.1427 16.1562 0.0183005 0.032101 2022-03-08T00:41:15.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:41:15.862173+0000 min lat: 0.0015392 max lat: 8.64344 avg lat: 0.0319795 2022-03-08T00:41:15.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:41:15.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 80 16 40035 40019 31.2604 40.5625 0.0211228 0.0319795 2022-03-08T00:41:16.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 81 16 40560 40544 31.2795 32.8125 0.0258312 0.0319512 2022-03-08T00:41:17.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 82 16 40972 40956 31.212 25.75 0.0293502 0.0319814 2022-03-08T00:41:18.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 83 16 41453 41437 31.1982 30.0625 0.0409301 0.0320439 2022-03-08T00:41:19.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 84 16 42066 42050 31.2828 38.3125 0.0352969 0.0319534 2022-03-08T00:41:20.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 85 16 42675 42659 31.3625 38.0625 0.0318773 0.0318761 2022-03-08T00:41:21.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 86 16 43164 43148 31.3532 30.5625 0.0252322 0.031878 2022-03-08T00:41:22.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 87 16 43690 43674 31.3706 32.875 0.0158933 0.0318654 2022-03-08T00:41:23.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 88 16 43846 43830 31.1249 9.75 0.0331197 0.0321183 2022-03-08T00:41:24.862 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 89 16 44370 44354 31.1432 32.75 0.0150247 0.0320916 2022-03-08T00:41:25.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 90 16 44847 44831 31.1283 29.8125 0.0215735 0.0321154 2022-03-08T00:41:26.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 91 16 45246 45230 31.0603 24.9375 0.0242585 0.0321852 2022-03-08T00:41:27.865 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 92 16 45885 45869 31.1567 39.9375 0.0235331 0.0320855 2022-03-08T00:41:28.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 93 16 46461 46445 31.2088 36 0.0460011 0.0320318 2022-03-08T00:41:29.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 94 16 47105 47089 31.3049 40.25 0.0223791 0.0319344 2022-03-08T00:41:30.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 95 16 47538 47522 31.2602 27.0625 0.00424513 0.031854 2022-03-08T00:41:31.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 96 16 47538 47522 30.9346 0 - 0.031854 2022-03-08T00:41:32.141 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:41:32.142 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:41:32.143 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:41:32.143 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:41:32.143 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:41:32.143 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:41:32.573 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:41:32.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 97 16 47538 47522 30.6157 0 - 0.031854 2022-03-08T00:41:33.353 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:41:33.353 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:41:33.863 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 98 16 47538 47522 30.3033 0 - 0.031854 2022-03-08T00:41:34.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 99 16 47538 47522 29.9972 0 - 0.031854 2022-03-08T00:41:35.270 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.269639 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:41:35.270 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.269653 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:41:35.271 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.269661 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:41:35.275 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.274574 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:41:35.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:41:35.864182+0000 min lat: 0.0015392 max lat: 8.64344 avg lat: 0.031854 2022-03-08T00:41:35.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:41:35.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 100 16 47538 47522 29.6972 0 - 0.031854 2022-03-08T00:41:35.960 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.960144 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:41:35.963 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:35.960375 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:41:36.411 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.411011 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:41:36.662 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.661166 I | op-mon: start running mons 2022-03-08T00:41:36.689 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.688966 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:41:36.728 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.727017 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:41:36.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.731796 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:41:36.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:36.731955 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:41:36.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 101 16 47538 47522 29.4032 0 - 0.031854 2022-03-08T00:41:37.532 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:37.532075 I | op-mon: targeting the mon count 1 2022-03-08T00:41:37.537 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:37.537134 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:41:37.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 102 16 47538 47522 29.115 0 - 0.031854 2022-03-08T00:41:37.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:37.910276 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:41:37.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:37.910302 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:41:38.244 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.243643 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:41:38.244 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.243666 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:41:38.612 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.611480 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:41:38.612 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.611511 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:41:38.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 103 16 47538 47522 28.8323 0 - 0.031854 2022-03-08T00:41:38.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.978727 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:41:38.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:38.978756 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:41:39.324 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.324213 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:41:39.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.324241 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T00:41:39.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.672473 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T00:41:39.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.672496 I | op-mon: checking for basic quorum with existing mons 2022-03-08T00:41:39.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 104 16 47538 47522 28.5551 0 - 0.031854 2022-03-08T00:41:39.901 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.901239 I | op-mon: mon "a" endpoint is [v2:10.98.3.151:3300,v1:10.98.3.151:6789] 2022-03-08T00:41:39.922 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.922017 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:41:39.947 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.946345 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:41:39.947 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:39.946588 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:41:40.087 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:40.087459 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T00:41:40.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:40.097580 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T00:41:40.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:40.097615 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:41:40.290 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:40.289298 I | op-mon: mons running: [a] 2022-03-08T00:41:40.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 105 16 47875 47859 28.4837 2.10625 0.0188379 0.0350942 2022-03-08T00:41:41.030 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:41.030580 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:41:41.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:41.030605 I | op-mon: mons created: 1 2022-03-08T00:41:41.834 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:41.832107 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:41:41.845 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:41.841187 I | op-mon: mons running: [a] 2022-03-08T00:41:41.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 106 16 48397 48381 28.5228 32.625 0.0389167 0.0350499 2022-03-08T00:41:42.669 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:42.669315 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:41:42.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:42.669850 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T00:41:42.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 107 16 48946 48930 28.5768 34.3125 0.024298 0.034984 2022-03-08T00:41:43.420 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:43.418507 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T00:41:43.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 108 16 49498 49482 28.6317 34.5 0.0151544 0.0349187 2022-03-08T00:41:44.159 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:44.158857 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T00:41:44.865 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 109 16 50049 50033 28.6849 34.4375 0.0231958 0.0348526 2022-03-08T00:41:44.958 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:44.944136 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T00:41:45.807 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:45.806778 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T00:41:45.807 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:45.806804 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T00:41:45.864 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 110 16 50600 50584 28.7371 34.4375 0.0135289 0.0347869 2022-03-08T00:41:46.577 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:46.577206 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T00:41:46.865 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 111 16 51113 51097 28.7671 32.0625 0.0275852 0.0347531 2022-03-08T00:41:47.331 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:47.330480 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T00:41:47.331 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:47.330557 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T00:41:47.332 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:47.330578 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T00:41:47.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 112 16 51635 51619 28.8013 32.625 0.0384882 0.0347108 2022-03-08T00:41:48.159 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:48.150506 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T00:41:48.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:48.187017 I | op-mgr: start running mgr 2022-03-08T00:41:48.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:48.187046 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T00:41:48.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 113 16 52151 52135 28.8318 32.25 0.0621543 0.0346738 2022-03-08T00:41:49.041 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.039300 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T00:41:49.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.047144 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T00:41:49.052 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.052353 I | op-mgr: setting services to point to mgr "a" 2022-03-08T00:41:49.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.191143 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T00:41:49.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.191169 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T00:41:49.219 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.219111 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T00:41:49.224 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223629 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-2" 2022-03-08T00:41:49.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223661 I | op-osd: OSD PVC "smithi120-lv-2-data-0twdg6" already exists 2022-03-08T00:41:49.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223672 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-3" 2022-03-08T00:41:49.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223681 I | op-osd: OSD PVC "smithi120-lv-3-data-0q4xnx" already exists 2022-03-08T00:41:49.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223688 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-4" 2022-03-08T00:41:49.228 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223705 I | op-osd: OSD PVC "smithi120-lv-4-data-0vbrdz" already exists 2022-03-08T00:41:49.228 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.223714 I | op-osd: creating 1 new PVCs for device set "smithi120-lv-1" 2022-03-08T00:41:49.235 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.232172 I | op-osd: successfully provisioned PVC "smithi120-lv-1-data-0c69wh" 2022-03-08T00:41:49.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.252326 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:41:49.253 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.252343 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:41:49.253 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.252347 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:41:49.253 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.252350 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0c69wh" 2022-03-08T00:41:49.340 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.339658 I | op-osd: started OSD provisioning job for PVC "smithi120-lv-1-data-0c69wh" 2022-03-08T00:41:49.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.339684 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T00:41:49.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.339689 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T00:41:49.467 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:49.465935 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0c69wh is "starting" 2022-03-08T00:41:49.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 114 16 52512 52496 28.7768 22.5625 0.0291836 0.0347339 2022-03-08T00:41:50.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 115 16 52637 52621 28.5945 7.8125 0.0171128 0.0347639 2022-03-08T00:41:51.304 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.304612 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:41:51.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.327607 I | op-osd: updating OSD 1 on PVC "smithi120-lv-2-data-0twdg6" 2022-03-08T00:41:51.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.327631 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:41:51.513 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.426003 I | op-mgr: successful modules: prometheus 2022-03-08T00:41:51.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.426139 I | op-mgr: successful modules: dashboard 2022-03-08T00:41:51.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:51.426882 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T00:41:51.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 116 16 52637 52621 28.348 0 - 0.0347639 2022-03-08T00:41:52.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 117 16 53110 53094 28.3584 14.7812 0.0254044 0.0352514 2022-03-08T00:41:53.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:53.578063 I | op-mgr: successful modules: balancer 2022-03-08T00:41:53.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 118 16 53136 53120 28.1318 1.625 0.0247519 0.0352762 2022-03-08T00:41:54.616 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:54.616519 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:41:54.839 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:54.831565 I | op-osd: updating OSD 2 on PVC "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:41:54.840 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:54.831588 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:41:54.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 119 16 53258 53242 27.9595 7.625 0.01196 0.0357566 2022-03-08T00:41:55.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:41:55.866738+0000 min lat: 0.0015392 max lat: 10.0377 avg lat: 0.0357094 2022-03-08T00:41:55.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:41:55.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 120 16 53752 53736 27.9838 30.875 0.0201962 0.0357094 2022-03-08T00:41:56.158 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:56.158119 I | op-mgr: successful modules: orchestrator modules 2022-03-08T00:41:56.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 121 16 54243 54227 28.0061 30.6875 0.0169354 0.0356967 2022-03-08T00:41:57.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 122 16 54753 54737 28.0378 31.875 0.0150366 0.0356569 2022-03-08T00:41:58.547 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:58.544607 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:41:58.551 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:58.550912 I | op-osd: updating OSD 3 on PVC "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:41:58.552 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:41:58.550936 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:41:58.866 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 123 16 55324 55308 28.0999 35.6875 0.0261372 0.0355797 2022-03-08T00:41:59.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 124 16 55847 55831 28.1369 32.6875 0.0249245 0.0355287 2022-03-08T00:42:00.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 125 16 56376 56360 28.1763 33.0625 0.0313375 0.0354813 2022-03-08T00:42:01.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 126 16 56904 56888 28.2145 33 0.0194721 0.0354197 2022-03-08T00:42:02.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 127 16 57346 57330 28.2098 27.625 0.0316531 0.03544 2022-03-08T00:42:03.354 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:42:03.355 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:42:03.355 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:42:03.355 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:42:03.356 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:42:03.356 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:42:03.854 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:42:03.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 128 16 57823 57807 28.2223 29.8125 0.0237596 0.0354235 2022-03-08T00:42:04.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 129 16 58282 58266 28.2259 28.6875 0.024151 0.0354193 2022-03-08T00:42:04.986 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:42:04.986 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:42:05.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 130 16 58666 58650 28.1934 24 0.0125148 0.0354433 2022-03-08T00:42:06.867 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 131 16 59170 59154 28.2186 31.5 0.0264614 0.0354272 2022-03-08T00:42:07.868 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 132 16 59621 59605 28.2184 28.1875 0.0262929 0.0354282 2022-03-08T00:42:08.868 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 133 16 60122 60106 28.2416 31.3125 0.0276099 0.0353992 2022-03-08T00:42:09.868 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 134 16 60668 60652 28.2855 34.125 0.0402087 0.0353417 2022-03-08T00:42:10.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 135 16 61177 61161 28.3114 31.8125 0.015594 0.0353141 2022-03-08T00:42:11.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 136 16 61732 61716 28.3583 34.6875 0.0276234 0.0352549 2022-03-08T00:42:12.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 137 16 62294 62278 28.4076 35.125 0.031712 0.0351842 2022-03-08T00:42:13.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 138 16 62773 62757 28.4187 29.9375 0.0193229 0.0351788 2022-03-08T00:42:14.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 139 16 63327 63311 28.4633 34.625 0.0527935 0.0351225 2022-03-08T00:42:15.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:42:15.869671+0000 min lat: 0.0015392 max lat: 10.0377 avg lat: 0.0351331 2022-03-08T00:42:15.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:42:15.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 140 16 63769 63753 28.4573 27.625 0.0330373 0.0351331 2022-03-08T00:42:16.869 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 141 16 64107 64091 28.4053 21.125 0.0204321 0.0351936 2022-03-08T00:42:17.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 142 16 64570 64554 28.4089 28.9375 0.007452 0.0351887 2022-03-08T00:42:18.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:18.856909 I | op-osd: OSD orchestration status for node smithi120-lv-1-data-0c69wh is "orchestrating" 2022-03-08T00:42:18.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 143 16 65032 65016 28.4121 28.875 0.0266884 0.0351869 2022-03-08T00:42:19.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:19.559392 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0c69wh is "orchestrating" 2022-03-08T00:42:19.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 144 16 65505 65489 28.4201 29.5625 0.0221117 0.0351773 2022-03-08T00:42:20.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 145 16 65925 65909 28.4051 26.25 0.0183014 0.0351949 2022-03-08T00:42:21.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 146 16 66405 66389 28.416 30 0.0347159 0.0351828 2022-03-08T00:42:22.870 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 147 16 66882 66866 28.4255 29.8125 0.0357369 0.0351687 2022-03-08T00:42:23.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 148 16 67278 67262 28.4006 24.75 0.0245429 0.0351999 2022-03-08T00:42:24.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 149 16 67697 67681 28.3858 26.1875 0.0458789 0.0352181 2022-03-08T00:42:25.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 150 16 68082 68066 28.3569 24.0625 0.0408061 0.0352571 2022-03-08T00:42:26.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 151 16 68589 68573 28.379 31.6875 0.038312 0.0352282 2022-03-08T00:42:27.782 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:27.780828 I | op-osd: OSD orchestration status for PVC smithi120-lv-1-data-0c69wh is "completed" 2022-03-08T00:42:27.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:27.780852 I | op-osd: creating OSD 0 on PVC "smithi120-lv-1-data-0c69wh" 2022-03-08T00:42:27.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:27.780862 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0c69wh" 2022-03-08T00:42:27.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 152 16 69066 69050 28.3884 29.8125 0.0130502 0.0352184 2022-03-08T00:42:28.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 153 16 69492 69476 28.3768 26.625 0.0248326 0.0352305 2022-03-08T00:42:29.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 154 16 69603 69587 28.2376 6.9375 0.034736 0.0352469 2022-03-08T00:42:30.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 155 16 69773 69757 28.124 10.625 0.0437725 0.0355481 2022-03-08T00:42:31.262 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:31.180187 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T00:42:31.270 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:31.180204 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:42:31.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:31.506551 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:42:31.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 156 16 70176 70160 28.1052 25.1875 0.0213626 0.0355707 2022-03-08T00:42:32.861 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:32.861338 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:42:32.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 157 16 70465 70449 28.0412 18.0625 0.0101238 0.0356455 2022-03-08T00:42:33.331 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:33.330590 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T00:42:33.871 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 158 16 70637 70621 27.9317 10.75 0.00360236 0.0356332 2022-03-08T00:42:34.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 159 16 70637 70621 27.7561 0 - 0.0356332 2022-03-08T00:42:34.987 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:42:34.988 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:42:34.989 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:42:34.989 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:42:34.990 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:42:34.991 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:42:35.432 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:42:35.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:42:35.872264+0000 min lat: 0.0015392 max lat: 10.0377 avg lat: 0.0356332 2022-03-08T00:42:35.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:42:35.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 160 16 70637 70621 27.5826 0 - 0.0356332 2022-03-08T00:42:36.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 161 16 70637 70621 27.4113 0 - 0.0356332 2022-03-08T00:42:37.015 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:42:37.015 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:42:37.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 162 16 70637 70621 27.2421 0 - 0.0356332 2022-03-08T00:42:38.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 163 16 70637 70621 27.075 0 - 0.0356332 2022-03-08T00:42:39.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 164 16 70637 70621 26.9099 0 - 0.0356332 2022-03-08T00:42:40.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 165 16 70637 70621 26.7468 0 - 0.0356332 2022-03-08T00:42:41.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 166 16 70637 70621 26.5857 0 - 0.0356332 2022-03-08T00:42:42.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 167 16 70637 70621 26.4265 0 - 0.0356332 2022-03-08T00:42:43.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 168 16 70637 70621 26.2692 0 - 0.0356332 2022-03-08T00:42:44.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 169 16 70637 70621 26.1138 0 - 0.0356332 2022-03-08T00:42:45.872 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 170 16 70637 70621 25.9602 0 - 0.0356332 2022-03-08T00:42:46.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 171 16 70760 70744 25.8533 0.591346 0.00637861 0.0385047 2022-03-08T00:42:47.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 172 16 70760 70744 25.703 0 - 0.0385047 2022-03-08T00:42:48.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 173 16 70760 70744 25.5544 0 - 0.0385047 2022-03-08T00:42:49.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 174 16 70760 70744 25.4076 0 - 0.0385047 2022-03-08T00:42:50.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 175 16 70760 70744 25.2624 0 - 0.0385047 2022-03-08T00:42:51.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 176 16 70760 70744 25.1189 0 - 0.0385047 2022-03-08T00:42:52.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 177 16 70885 70869 25.0211 1.30208 0.00417679 0.0398872 2022-03-08T00:42:53.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 178 16 70885 70869 24.8805 0 - 0.0398872 2022-03-08T00:42:54.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 179 16 70885 70869 24.7415 0 - 0.0398872 2022-03-08T00:42:55.873 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:42:55.874186+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0398872 2022-03-08T00:42:55.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:42:55.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 180 16 70885 70869 24.6041 0 - 0.0398872 2022-03-08T00:42:56.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 181 16 70885 70869 24.4681 0 - 0.0398872 2022-03-08T00:42:57.170 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.170184 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T00:42:57.171 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.170207 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T00:42:57.171 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.170215 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T00:42:57.485 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.485478 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:42:57.488 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.488922 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:42:57.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:57.489127 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:42:57.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 182 16 70885 70869 24.3337 0 - 0.0398872 2022-03-08T00:42:58.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.009709 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T00:42:58.151 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.150717 I | op-mon: start running mons 2022-03-08T00:42:58.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.206144 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:42:58.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.288509 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:42:58.295 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.295695 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:42:58.296 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:58.295817 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:42:58.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 183 16 70885 70869 24.2007 0 - 0.0398872 2022-03-08T00:42:59.112 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:59.112134 I | op-mon: targeting the mon count 1 2022-03-08T00:42:59.115 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:59.115227 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:42:59.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:59.641772 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T00:42:59.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:42:59.641798 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:42:59.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 184 16 70885 70869 24.0692 0 - 0.0398872 2022-03-08T00:43:00.016 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:00.011864 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T00:43:00.016 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:00.011889 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:43:00.403 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:00.402990 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T00:43:00.403 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:00.403125 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:43:00.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 185 16 70923 70907 23.952 0.296875 0.00420821 0.0406326 2022-03-08T00:43:01.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.076955 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T00:43:01.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.076979 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:43:01.513 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.513351 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T00:43:01.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.513385 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T00:43:01.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 186 16 70923 70907 23.8232 0 - 0.0406326 2022-03-08T00:43:02.059 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.886335 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T00:43:02.060 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:01.886364 I | op-mon: checking for basic quorum with existing mons 2022-03-08T00:43:02.526 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.525682 I | op-mon: mon "a" endpoint is [v2:10.98.3.151:3300,v1:10.98.3.151:6789] 2022-03-08T00:43:02.671 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.670666 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.3.151:6789"]}] data:a=10.98.3.151:6789 mapping:{"node":{"a":{"Name":"smithi120","Hostname":"smithi120","Address":"172.21.15.120"}}} maxMonId:0] 2022-03-08T00:43:02.743 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.743857 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T00:43:02.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.744028 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T00:43:02.803 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.803100 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T00:43:02.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.810035 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T00:43:02.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.810063 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:43:02.815 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:02.815516 I | op-mon: mons running: [a] 2022-03-08T00:43:02.874 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 187 16 71010 70994 23.7249 2.71875 0.00421392 0.0419252 2022-03-08T00:43:03.291 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:03.291500 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:43:03.292 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:03.291528 I | op-mon: mons created: 1 2022-03-08T00:43:03.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 188 16 71010 70994 23.5987 0 - 0.0419252 2022-03-08T00:43:03.922 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:03.922656 I | op-mon: waiting for mon quorum with [a] 2022-03-08T00:43:04.019 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:04.017869 I | op-mon: mons running: [a] 2022-03-08T00:43:04.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:04.521088 I | op-mon: Monitors in quorum: [a] 2022-03-08T00:43:04.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:04.521127 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T00:43:04.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 189 16 71010 70994 23.4738 0 - 0.0419252 2022-03-08T00:43:04.959 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:04.959403 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T00:43:05.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:05.437948 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T00:43:05.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 190 16 71010 70994 23.3503 0 - 0.0419252 2022-03-08T00:43:05.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:05.900247 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T00:43:06.340 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:06.340039 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T00:43:06.340 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:06.340063 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T00:43:06.770 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:06.770602 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T00:43:06.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 191 16 71010 70994 23.228 0 - 0.0419252 2022-03-08T00:43:07.016 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:43:07.016 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:43:07.017 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:43:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:43:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:43:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:43:07.110 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:43:07.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.307633 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T00:43:07.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.307657 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T00:43:07.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.307661 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T00:43:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=3 2022-03-08T00:43:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:43:07.751 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.750874 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T00:43:07.778 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.777784 I | op-mgr: start running mgr 2022-03-08T00:43:07.778 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:07.777815 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T00:43:07.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 192 16 71010 70994 23.107 0 - 0.0419252 2022-03-08T00:43:08.303 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.302889 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T00:43:08.313 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.311833 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T00:43:08.316 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.314469 I | op-mgr: setting services to point to mgr "a" 2022-03-08T00:43:08.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.464892 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T00:43:08.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.464917 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T00:43:08.499 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.499194 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T00:43:08.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503159 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-2" 2022-03-08T00:43:08.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503183 I | op-osd: OSD PVC "smithi120-lv-2-data-0twdg6" already exists 2022-03-08T00:43:08.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503191 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-3" 2022-03-08T00:43:08.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503205 I | op-osd: OSD PVC "smithi120-lv-3-data-0q4xnx" already exists 2022-03-08T00:43:08.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503212 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-4" 2022-03-08T00:43:08.510 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503231 I | op-osd: OSD PVC "smithi120-lv-4-data-0vbrdz" already exists 2022-03-08T00:43:08.510 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503241 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi120-lv-1" 2022-03-08T00:43:08.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.503251 I | op-osd: OSD PVC "smithi120-lv-1-data-0c69wh" already exists 2022-03-08T00:43:08.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519785 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:43:08.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519912 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:43:08.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519922 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:43:08.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519927 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0c69wh" 2022-03-08T00:43:08.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519931 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T00:43:08.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:08.519934 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T00:43:08.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 193 16 71259 71243 23.0679 2.59375 0.0497457 0.0433397 2022-03-08T00:43:09.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 194 16 71627 71611 23.0676 23 0.0381688 0.043342 2022-03-08T00:43:10.037 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:10.035953 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T00:43:10.040 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:10.039829 I | op-mgr: successful modules: prometheus 2022-03-08T00:43:10.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:10.061692 I | op-mgr: successful modules: dashboard 2022-03-08T00:43:10.844 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:10.843253 I | op-mgr: successful modules: balancer 2022-03-08T00:43:10.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 195 16 72006 71990 23.0707 23.6875 0.0960095 0.0433319 2022-03-08T00:43:11.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:11.010233 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:43:11.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:11.046679 I | op-osd: updating OSD 0 on PVC "smithi120-lv-1-data-0c69wh" 2022-03-08T00:43:11.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:11.046704 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-1-data-0c69wh" 2022-03-08T00:43:11.596 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:11.596575 I | op-mgr: successful modules: orchestrator modules 2022-03-08T00:43:11.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 196 16 72082 72066 22.9773 4.75 0.0288912 0.0433301 2022-03-08T00:43:12.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:12.341532 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:43:12.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:12.349875 I | op-osd: updating OSD 1 on PVC "smithi120-lv-2-data-0twdg6" 2022-03-08T00:43:12.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:12.349905 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-2-data-0twdg6" 2022-03-08T00:43:12.875 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 197 16 72082 72066 22.8606 0 - 0.0433301 2022-03-08T00:43:13.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:13.629120 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:43:13.634 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:13.634775 I | op-osd: updating OSD 2 on PVC "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:43:13.635 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:13.634791 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-3-data-0q4xnx" 2022-03-08T00:43:13.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 198 16 72082 72066 22.7452 0 - 0.0433301 2022-03-08T00:43:14.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 199 16 72082 72066 22.6309 0 - 0.0433301 2022-03-08T00:43:15.447 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:15.446958 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T00:43:15.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:15.452903 I | op-osd: updating OSD 3 on PVC "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:43:15.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:15.452929 I | op-osd: OSD will have its main bluestore block on "smithi120-lv-4-data-0vbrdz" 2022-03-08T00:43:15.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:43:15.876364+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0433301 2022-03-08T00:43:15.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:43:15.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 200 16 72082 72066 22.5177 0 - 0.0433301 2022-03-08T00:43:16.330 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:16.329678 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T00:43:16.330 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:43:16.329703 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T00:43:16.876 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 201 16 72349 72333 22.4887 3.3375 0.0472796 0.0444568 2022-03-08T00:43:17.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 202 16 72664 72648 22.4748 19.6875 0.0436527 0.0444842 2022-03-08T00:43:18.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 203 16 72974 72958 22.4595 19.375 0.11967 0.0445113 2022-03-08T00:43:19.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 204 16 73233 73217 22.4287 16.1875 0.00632088 0.0445346 2022-03-08T00:43:20.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 205 16 73437 73421 22.3815 12.75 0.0255363 0.0446689 2022-03-08T00:43:21.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 206 16 73741 73725 22.3651 19 0.0217739 0.0446968 2022-03-08T00:43:22.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 207 16 74040 74024 22.3473 18.6875 0.0396931 0.0447348 2022-03-08T00:43:23.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 208 16 74352 74336 22.3336 19.5 0.024459 0.0447641 2022-03-08T00:43:24.877 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 209 16 74666 74650 22.3206 19.625 0.039619 0.0447927 2022-03-08T00:43:25.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 210 16 74952 74936 22.2995 17.875 0.0267504 0.0448314 2022-03-08T00:43:26.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 211 16 75205 75189 22.2687 15.8125 0.0369105 0.0448928 2022-03-08T00:43:27.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 212 16 75521 75505 22.2568 19.75 0.0850301 0.0449193 2022-03-08T00:43:28.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 213 16 75861 75845 22.2521 21.25 0.0289172 0.0449261 2022-03-08T00:43:29.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 214 16 76083 76067 22.2129 13.875 0.0122939 0.0449451 2022-03-08T00:43:30.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 215 16 76283 76267 22.1677 12.5 0.0101199 0.0450462 2022-03-08T00:43:31.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 216 16 76573 76557 22.149 18.125 0.0346969 0.045139 2022-03-08T00:43:32.878 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 217 16 76906 76890 22.1428 20.8125 0.0628715 0.0451504 2022-03-08T00:43:33.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 218 16 77238 77222 22.1364 20.75 0.0352246 0.0451634 2022-03-08T00:43:34.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 219 16 77552 77536 22.125 19.625 0.0240562 0.0451849 2022-03-08T00:43:35.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:43:35.879321+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0452143 2022-03-08T00:43:35.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:43:35.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 220 16 77859 77843 22.1116 19.1875 0.0366028 0.0452143 2022-03-08T00:43:36.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 221 16 78120 78104 22.0854 16.3125 0.037418 0.045265 2022-03-08T00:43:37.603 INFO:teuthology.orchestra.run.smithi120.stdout:waiting for osd to come back up 2022-03-08T00:43:37.604 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T00:43:37.604 INFO:teuthology.orchestra.run.smithi120.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T00:43:37.604 INFO:teuthology.orchestra.run.smithi120.stderr:++ toolbox ceph osd stat 2022-03-08T00:43:37.604 INFO:teuthology.orchestra.run.smithi120.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T00:43:37.605 INFO:teuthology.orchestra.run.smithi120.stderr:++ cut -f3 -d ' ' 2022-03-08T00:43:37.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 222 16 78346 78330 22.0495 14.125 0.0577713 0.0453413 2022-03-08T00:43:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T00:43:38.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 223 16 78650 78634 22.0358 19 0.0580535 0.0453656 2022-03-08T00:43:38.880 INFO:teuthology.orchestra.run.smithi120.stderr:+ num_osd=4 2022-03-08T00:43:38.880 INFO:teuthology.orchestra.run.smithi120.stderr:+ sleep 30s 2022-03-08T00:43:39.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 224 16 78959 78943 22.0237 19.3125 0.0510835 0.0453962 2022-03-08T00:43:40.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 225 16 79238 79222 22.0033 17.4375 0.048934 0.045438 2022-03-08T00:43:41.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 226 16 79333 79317 21.9322 5.9375 0.00643731 0.045445 2022-03-08T00:43:42.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 227 16 79333 79317 21.8356 0 - 0.045445 2022-03-08T00:43:43.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 228 16 79458 79442 21.7741 3.90625 0.0315611 0.0459153 2022-03-08T00:43:44.879 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 229 16 79788 79772 21.769 20.625 0.0331618 0.0459248 2022-03-08T00:43:45.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 230 16 80108 80092 21.7613 20 0.00950833 0.0459401 2022-03-08T00:43:46.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 231 16 80438 80422 21.7564 20.625 0.0289195 0.0459541 2022-03-08T00:43:47.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 232 16 80755 80739 21.748 19.8125 0.0452422 0.0459705 2022-03-08T00:43:48.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 233 16 81077 81061 21.741 20.125 0.0728201 0.0459831 2022-03-08T00:43:49.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 234 16 81413 81397 21.7379 21 0.0164947 0.0459874 2022-03-08T00:43:50.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 235 16 81742 81726 21.7329 20.5625 0.0229317 0.0460001 2022-03-08T00:43:51.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 236 16 82071 82055 21.7279 20.5625 0.0183759 0.0460138 2022-03-08T00:43:52.880 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 237 16 82404 82388 21.724 20.8125 0.018352 0.0460181 2022-03-08T00:43:53.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 238 16 82571 82555 21.6766 10.4375 0.0908787 0.0461209 2022-03-08T00:43:54.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 239 16 82909 82893 21.6743 21.125 0.0700284 0.0461278 2022-03-08T00:43:55.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:43:55.881295+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0461381 2022-03-08T00:43:55.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:43:55.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 240 16 83236 83220 21.6691 20.4375 0.030929 0.0461381 2022-03-08T00:43:56.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 241 16 83555 83539 21.6619 19.9375 0.100714 0.0461525 2022-03-08T00:43:57.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 242 16 83840 83824 21.646 17.8125 0.0151559 0.0461828 2022-03-08T00:43:58.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 243 16 84158 84142 21.6387 19.875 0.021108 0.0462014 2022-03-08T00:43:59.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 244 16 84470 84454 21.6299 19.5 0.0428702 0.0462203 2022-03-08T00:44:00.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 245 16 84779 84763 21.6205 19.3125 0.0733272 0.046241 2022-03-08T00:44:01.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 246 16 85098 85082 21.6136 19.9375 0.034288 0.0462557 2022-03-08T00:44:02.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 247 16 85363 85347 21.5932 16.5625 0.0545308 0.0462668 2022-03-08T00:44:03.881 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 248 16 85569 85553 21.558 12.875 0.00615426 0.0463499 2022-03-08T00:44:04.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 249 16 85756 85740 21.5183 11.6875 0.0349016 0.0464632 2022-03-08T00:44:05.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 250 16 86057 86041 21.5075 18.8125 0.0391947 0.0464836 2022-03-08T00:44:06.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 251 16 86373 86357 21.5005 19.75 0.0667614 0.0465003 2022-03-08T00:44:07.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 252 16 86688 86672 21.4933 19.6875 0.0871646 0.0465168 2022-03-08T00:44:08.881 INFO:teuthology.orchestra.run.smithi120.stderr:+ '[' 4 '!=' 4 ']' 2022-03-08T00:44:08.883 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T00:44:08.898 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch status 2022-03-08T00:44:08.899 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 253 16 87006 86990 21.4869 19.875 0.0910286 0.0465286 2022-03-08T00:44:09.882 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 254 16 87309 87293 21.4768 18.9375 0.0181808 0.0465475 2022-03-08T00:44:10.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 255 16 87525 87509 21.4456 13.5 0.00967323 0.0465578 2022-03-08T00:44:11.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 256 16 87814 87798 21.4323 18.0625 0.0458136 0.046647 2022-03-08T00:44:12.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 257 16 88132 88116 21.4263 19.875 0.0574864 0.0466626 2022-03-08T00:44:13.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 258 16 88404 88388 21.4091 17 0.0434128 0.0466974 2022-03-08T00:44:14.024 INFO:teuthology.orchestra.run.smithi120.stdout:Backend: rook 2022-03-08T00:44:14.024 INFO:teuthology.orchestra.run.smithi120.stdout:Available: Yes 2022-03-08T00:44:14.064 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ps 2022-03-08T00:44:14.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 259 16 88632 88616 21.3815 14.25 0.00744232 0.0467273 2022-03-08T00:44:15.075 INFO:teuthology.orchestra.run.smithi120.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T00:44:15.075 INFO:teuthology.orchestra.run.smithi120.stdout:crashcollector.crash smithi120 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T00:44:15.076 INFO:teuthology.orchestra.run.smithi120.stdout:mgr.a smithi120 running (9m) 0s ago 9m - - 4b871cde5e1a 2022-03-08T00:44:15.076 INFO:teuthology.orchestra.run.smithi120.stdout:mon.a smithi120 running (9m) 0s ago 9m - - 4b871cde5e1a 2022-03-08T00:44:15.076 INFO:teuthology.orchestra.run.smithi120.stdout:osd.0 smithi120 running (105s) 0s ago 106s - - 4b871cde5e1a 2022-03-08T00:44:15.076 INFO:teuthology.orchestra.run.smithi120.stdout:osd.1 smithi120 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T00:44:15.077 INFO:teuthology.orchestra.run.smithi120.stdout:osd.2 smithi120 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T00:44:15.077 INFO:teuthology.orchestra.run.smithi120.stdout:osd.3 smithi120 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T00:44:15.091 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls 2022-03-08T00:44:15.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:44:15.883674+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0467273 2022-03-08T00:44:15.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:44:15.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 260 16 88632 88616 21.2992 0 - 0.0467273 2022-03-08T00:44:16.530 INFO:teuthology.orchestra.run.smithi120.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T00:44:16.530 INFO:teuthology.orchestra.run.smithi120.stdout:crash 1/1 0s ago 7m * 2022-03-08T00:44:16.531 INFO:teuthology.orchestra.run.smithi120.stdout:mgr 1/1 0s ago 9m count:1 2022-03-08T00:44:16.531 INFO:teuthology.orchestra.run.smithi120.stdout:mon 1/1 0s ago 9m count:1 2022-03-08T00:44:16.531 INFO:teuthology.orchestra.run.smithi120.stdout:osd 8 0s ago 7m 2022-03-08T00:44:16.531 INFO:teuthology.orchestra.run.smithi120.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T00:44:16.558 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch host ls 2022-03-08T00:44:16.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 261 16 88822 88806 21.2631 5.9375 0.0263967 0.0470196 2022-03-08T00:44:17.665 INFO:teuthology.orchestra.run.smithi120.stdout:HOST ADDR LABELS STATUS 2022-03-08T00:44:17.665 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 172.21.15.120/smithi120 2022-03-08T00:44:17.666 INFO:teuthology.orchestra.run.smithi120.stdout:1 hosts in cluster 2022-03-08T00:44:17.704 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch device ls 2022-03-08T00:44:17.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 262 16 89050 89034 21.2363 14.25 0.0449341 0.0470798 2022-03-08T00:44:18.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 263 16 89327 89311 21.2214 17.3125 0.041514 0.0471107 2022-03-08T00:44:19.480 INFO:teuthology.orchestra.run.smithi120.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T00:44:19.481 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T00:44:19.481 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T00:44:19.481 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T00:44:19.481 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T00:44:19.496 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch apply rgw foo 2022-03-08T00:44:19.883 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 264 16 89576 89560 21.2 15.5625 0.0250635 0.0471549 2022-03-08T00:44:20.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 265 16 89814 89798 21.1761 14.875 0.0892216 0.0472109 2022-03-08T00:44:21.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:21.172954 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T00:44:21.176 INFO:teuthology.orchestra.run.smithi120.stdout:Created 2022-03-08T00:44:21.219 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch apply mds foo 2022-03-08T00:44:21.332 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:21.331639 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:44:21.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 266 16 90047 90031 21.1512 14.5625 0.0366213 0.0472652 2022-03-08T00:44:22.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:22.035250 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T00:44:22.036 INFO:teuthology.orchestra.run.smithi120.stdout:Created 2022-03-08T00:44:22.064 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch apply rbd-mirror 2022-03-08T00:44:22.065 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:22.064087 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:44:22.071 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:22.070393 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:44:22.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 267 16 90273 90257 21.1249 14.125 0.141727 0.0473262 2022-03-08T00:44:23.321 INFO:teuthology.orchestra.run.smithi120.stdout:Success 2022-03-08T00:44:23.355 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:23.351909 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:44:23.466 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch apply nfs foo --port 12777 2022-03-08T00:44:23.885 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 268 16 90449 90433 21.0871 11 0.0448844 0.0474028 2022-03-08T00:44:24.443 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.436021 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:44:24.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.462667 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T00:44:24.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.462693 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T00:44:24.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.462705 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T00:44:24.496 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.485757 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:44:24.497 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:24.485783 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:44:24.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 269 16 90621 90605 21.0487 10.75 0.0439387 0.0474967 2022-03-08T00:44:25.885 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 270 16 90859 90843 21.0258 14.875 0.0449296 0.0475479 2022-03-08T00:44:26.140 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:26.136568 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:44:26.314 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:26.307323 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T00:44:26.685 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:26.684956 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi120": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:44:26.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 271 16 91023 91007 20.986 10.25 0.239034 0.0476282 2022-03-08T00:44:27.776 INFO:teuthology.orchestra.run.smithi120.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T00:44:27.777 INFO:teuthology.orchestra.run.smithi120.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T00:44:27.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:27.760849 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi120": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:44:27.830 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-08T00:44:27.838 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-08T00:44:27.838 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-08T00:44:27.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 272 16 91213 91197 20.9525 11.875 0.155646 0.0477106 2022-03-08T00:44:28.884 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 273 16 91367 91351 20.911 9.625 0.0673787 0.0478075 2022-03-08T00:44:29.885 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 274 16 91482 91466 20.861 7.1875 0.0399791 0.0479219 2022-03-08T00:44:30.886 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 275 16 91736 91720 20.8428 15.875 0.134045 0.0479704 2022-03-08T00:44:31.886 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 276 16 92024 92008 20.8324 18 0.0410152 0.047992 2022-03-08T00:44:31.897 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:31.896590 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T00:44:32.886 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 277 16 92271 92255 20.813 15.4375 0.0350702 0.0480348 2022-03-08T00:44:33.886 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 278 16 92513 92497 20.7925 15.125 0.0360559 0.048081 2022-03-08T00:44:34.886 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 279 16 92678 92662 20.7549 10.3125 0.0461459 0.0481679 2022-03-08T00:44:35.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:44:35.887181+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0482356 2022-03-08T00:44:35.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:44:35.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 280 15 92883 92868 20.7268 12.875 0.129725 0.0482356 2022-03-08T00:44:36.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 281 16 93134 93118 20.7086 15.625 0.0374197 0.0482785 2022-03-08T00:44:37.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 282 16 93284 93268 20.6684 9.375 0.075494 0.0483228 2022-03-08T00:44:38.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 283 16 93526 93510 20.6488 15.125 0.0431066 0.0484134 2022-03-08T00:44:39.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 284 16 93820 93804 20.6408 18.375 0.0505851 0.0484383 2022-03-08T00:44:40.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 285 16 94041 94025 20.6168 13.8125 0.13808 0.0484889 2022-03-08T00:44:41.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 286 16 94258 94242 20.5922 13.5625 0.0525046 0.0485522 2022-03-08T00:44:42.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 287 16 94520 94504 20.5775 16.375 0.029229 0.0485861 2022-03-08T00:44:42.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:42.992690 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-08T00:44:43.020 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:43.016454 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-08T00:44:43.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 288 16 94724 94708 20.5503 12.75 0.0846419 0.0486495 2022-03-08T00:44:44.112 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:44.112611 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-08T00:44:44.123 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:44.123260 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-08T00:44:44.135 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:44.131606 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-08T00:44:44.152 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:44.151636 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-08T00:44:44.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 289 16 94904 94888 20.5181 11.25 0.115425 0.0487116 2022-03-08T00:44:45.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 290 16 95139 95123 20.498 14.6875 0.0828682 0.0487731 2022-03-08T00:44:46.887 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 291 16 95413 95397 20.4864 17.125 0.039821 0.0488033 2022-03-08T00:44:46.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:46.977082 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T00:44:46.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:46.983887 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-08T00:44:47.004 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:47.004147 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-08T00:44:47.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 292 16 95613 95597 20.459 12.5 0.100811 0.0488606 2022-03-08T00:44:48.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 293 16 95875 95859 20.4451 16.375 0.081852 0.0489012 2022-03-08T00:44:49.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:49.597476 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-08T00:44:49.648 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:49.632062 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-08T00:44:49.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:49.643497 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-08T00:44:49.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 294 16 96114 96098 20.4264 14.9375 0.140108 0.0489355 2022-03-08T00:44:50.333 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:50.331976 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:44:50.333 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:50.332002 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:44:50.537 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:50.536685 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:44:50.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 295 16 96372 96356 20.4118 16.125 0.0629986 0.0489754 2022-03-08T00:44:51.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 296 16 96577 96561 20.3861 12.8125 0.0809829 0.0490418 2022-03-08T00:44:52.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 297 16 96823 96807 20.3692 15.375 0.0413983 0.0490822 2022-03-08T00:44:53.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 298 16 97050 97034 20.3485 14.1875 0.0268155 0.0491309 2022-03-08T00:44:54.136 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:54.136079 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:44:54.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:54.238557 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:44:54.239 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:54.238577 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:44:54.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:54.453044 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-08T00:44:54.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:54.847633 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:44:54.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 299 16 97300 97284 20.3327 15.625 0.0490236 0.0491654 2022-03-08T00:44:55.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:44:55.889076+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0492592 2022-03-08T00:44:55.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:44:55.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 300 16 97413 97397 20.2884 7.0625 0.507663 0.0492592 2022-03-08T00:44:56.888 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 301 16 97650 97634 20.2703 14.8125 0.0445296 0.0493206 2022-03-08T00:44:57.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:57.505336 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-08T00:44:57.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:57.505369 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T00:44:57.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 302 16 97925 97909 20.26 17.1875 0.0436686 0.0493459 2022-03-08T00:44:58.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 303 16 98165 98149 20.2426 15 0.102777 0.04939 2022-03-08T00:44:59.310 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:59.310206 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:44:59.325 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:44:59.325676 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:44:59.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 304 16 98408 98392 20.226 15.1875 0.0154262 0.0494286 2022-03-08T00:45:00.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 305 16 98664 98648 20.2121 16 0.0569763 0.049462 2022-03-08T00:45:00.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:00.900564 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T00:45:01.753 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:01.753487 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T00:45:01.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 306 16 98890 98874 20.1922 14.125 0.127349 0.0495098 2022-03-08T00:45:02.336 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:02.325142 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:02.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:02.474545 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:02.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:02.474563 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:02.889 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 307 16 99147 99131 20.1788 16.0625 0.0271447 0.0495456 2022-03-08T00:45:03.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:03.036569 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:03.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:03.642107 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T00:45:03.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 308 16 99336 99320 20.1516 11.8125 0.022249 0.0496099 2022-03-08T00:45:04.689 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:04.689018 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T00:45:04.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 309 16 99541 99525 20.1279 12.8125 0.0611516 0.0496705 2022-03-08T00:45:05.632 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:05.631887 I | op-mds: setting mds config flags 2022-03-08T00:45:05.637 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:05.632664 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T00:45:05.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 310 16 99782 99766 20.1115 15.0625 0.0636167 0.0497106 2022-03-08T00:45:06.246 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:06.245798 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:06.271 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:06.270857 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:06.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:06.501246 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T00:45:06.516 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:06.509360 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T00:45:06.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 311 16 100002 99986 20.0911 13.75 0.0553158 0.0497623 2022-03-08T00:45:07.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 312 16 100108 100092 20.0479 6.625 0.0141233 0.0498001 2022-03-08T00:45:08.303 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:08.303556 I | op-mds: setting mds config flags 2022-03-08T00:45:08.304 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:08.303585 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T00:45:08.626 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:08.536059 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:08.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 313 16 100317 100301 20.0256 13.0625 0.0670974 0.0499259 2022-03-08T00:45:09.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:09.799263 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T00:45:09.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 314 16 100557 100541 20.0096 15 0.0202204 0.0499653 2022-03-08T00:45:10.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 315 16 100828 100812 19.9998 16.9375 0.0643085 0.0499931 2022-03-08T00:45:11.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:11.088542 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:11.091 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:11.088567 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:11.789 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:11.788581 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:11.890 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 316 16 101060 101044 19.9824 14.5 0.198983 0.050026 2022-03-08T00:45:12.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 317 16 101242 101226 19.9552 11.375 0.0174672 0.0500981 2022-03-08T00:45:13.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 318 16 101476 101460 19.9385 14.625 0.0595484 0.0501359 2022-03-08T00:45:14.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 319 16 101739 101723 19.9275 16.4375 0.0239549 0.0501739 2022-03-08T00:45:14.994 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:14.992211 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:15.050 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:15.045062 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:15.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:45:15.891621+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0502193 2022-03-08T00:45:15.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:45:15.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 320 16 101938 101922 19.9041 12.4375 0.106096 0.0502193 2022-03-08T00:45:16.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 321 16 102200 102184 19.8931 16.375 0.0203921 0.0502571 2022-03-08T00:45:17.603 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:17.599725 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:17.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 322 16 102455 102439 19.8808 15.9375 0.0246878 0.05029 2022-03-08T00:45:18.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 323 16 102712 102696 19.869 16.0625 0.029523 0.0503184 2022-03-08T00:45:19.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 324 16 103012 102996 19.8655 18.75 0.0463162 0.0503298 2022-03-08T00:45:19.995 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:19.994133 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:19.995 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:19.994158 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:20.763 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:20.762785 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:20.891 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 325 16 103219 103203 19.8442 12.9375 0.0507168 0.0503768 2022-03-08T00:45:21.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 326 16 103395 103379 19.8171 11 0.0115571 0.0504497 2022-03-08T00:45:22.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 327 16 103644 103628 19.8041 15.5625 0.113097 0.0504819 2022-03-08T00:45:23.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 328 16 103861 103845 19.785 13.5625 0.0777608 0.0505337 2022-03-08T00:45:24.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 329 16 104165 104149 19.7826 19 0.0568341 0.0505388 2022-03-08T00:45:25.015 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:25.012600 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:25.400 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:25.399482 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:25.892 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 330 16 104428 104412 19.7725 16.4375 0.0594805 0.0505658 2022-03-08T00:45:26.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 331 16 104672 104656 19.7588 15.25 0.0869099 0.0506007 2022-03-08T00:45:27.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 332 16 104946 104930 19.7508 17.125 0.0798953 0.0506195 2022-03-08T00:45:28.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 333 16 105193 105177 19.7379 15.4375 0.0414744 0.0506525 2022-03-08T00:45:29.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:29.680124 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:29.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 334 16 105435 105419 19.7241 15.125 0.0183163 0.0506868 2022-03-08T00:45:30.166 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:30.164807 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:30.166 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:30.164831 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:30.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 335 16 105693 105677 19.7133 16.125 0.03335 0.0507164 2022-03-08T00:45:30.897 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:30.892803 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:31.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 336 16 105869 105853 19.6874 11 0.100834 0.050777 2022-03-08T00:45:32.893 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 337 16 106105 106089 19.6727 14.75 0.0821585 0.0508188 2022-03-08T00:45:33.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 338 16 106362 106346 19.662 16.0625 0.0685846 0.0508489 2022-03-08T00:45:34.041 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:34.041109 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:34.209 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:34.208924 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:34.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 339 16 106617 106601 19.651 15.9375 0.0296556 0.0508785 2022-03-08T00:45:35.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:45:35.894484+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0509062 2022-03-08T00:45:35.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:45:35.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 340 16 106863 106847 19.6385 15.375 0.0712852 0.0509062 2022-03-08T00:45:36.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:36.464622 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:36.566 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:36.565542 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:36.566 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:36.565564 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:36.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 341 16 107126 107110 19.6291 16.4375 0.017796 0.050936 2022-03-08T00:45:37.179 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:37.176228 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:37.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 342 16 107262 107246 19.5965 8.5 0.120336 0.0510165 2022-03-08T00:45:38.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 343 16 107494 107478 19.5817 14.5 0.0444812 0.0510584 2022-03-08T00:45:39.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 344 16 107622 107606 19.548 8 0.0118256 0.0511246 2022-03-08T00:45:40.251 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:40.251452 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:40.577 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:40.577421 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:40.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 345 16 107834 107818 19.5297 13.25 0.0324102 0.0511949 2022-03-08T00:45:41.894 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 346 16 108029 108013 19.5085 12.1875 0.138906 0.0512495 2022-03-08T00:45:42.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 347 16 108230 108214 19.4885 12.5625 0.0489302 0.0512413 2022-03-08T00:45:42.955 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:42.954893 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:43.419 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:43.412765 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:43.419 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:43.412788 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:43.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 348 16 108411 108395 19.465 11.3125 0.0588675 0.0513648 2022-03-08T00:45:44.020 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:44.020394 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:44.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 349 16 108649 108633 19.4518 14.875 0.170138 0.0513969 2022-03-08T00:45:45.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 350 16 108928 108912 19.4461 17.4375 0.0307285 0.0514114 2022-03-08T00:45:46.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 351 16 109288 109272 19.4548 22.5 0.0300867 0.0513901 2022-03-08T00:45:47.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:47.494049 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:47.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 352 16 109666 109650 19.4666 23.625 0.0128272 0.0513604 2022-03-08T00:45:48.142 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:48.141492 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:48.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 353 16 110035 110019 19.4768 23.0625 0.0318405 0.0513325 2022-03-08T00:45:49.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 354 16 110414 110398 19.4887 23.6875 0.0675419 0.0513019 2022-03-08T00:45:50.466 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:50.465852 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:50.567 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:50.566048 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:50.567 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:50.566074 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:50.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 355 16 110779 110763 19.498 22.8125 0.0254428 0.051277 2022-03-08T00:45:51.276 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:51.276529 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:45:51.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 356 16 111039 111023 19.4889 16.25 0.0154168 0.0512948 2022-03-08T00:45:52.895 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 357 16 111249 111233 19.4711 13.125 0.0566327 0.0513493 2022-03-08T00:45:53.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 358 16 111617 111601 19.4809 23 0.0242743 0.0513223 2022-03-08T00:45:54.087 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:54.086440 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:45:54.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 359 16 111919 111903 19.4792 18.875 0.020037 0.0512922 2022-03-08T00:45:55.385 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:55.384724 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:45:55.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:2022-03-08T00:45:55.896598+0000 min lat: 0.0015392 max lat: 13.0063 avg lat: 0.0513588 2022-03-08T00:45:55.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T00:45:55.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout: 360 15 112148 112133 19.4651 14.375 0.0196393 0.0513588 2022-03-08T00:45:56.839 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:56.839746 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:45:56.896 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Total time run: 360.415 2022-03-08T00:45:56.897 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Total writes made: 112148 2022-03-08T00:45:56.897 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Write size: 65536 2022-03-08T00:45:56.897 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Object size: 65536 2022-03-08T00:45:56.897 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Bandwidth (MB/sec): 19.4477 2022-03-08T00:45:56.897 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Stddev Bandwidth: 15.3792 2022-03-08T00:45:56.898 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Max bandwidth (MB/sec): 92.625 2022-03-08T00:45:56.898 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Min bandwidth (MB/sec): 0 2022-03-08T00:45:56.898 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Average IOPS: 311 2022-03-08T00:45:56.898 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Stddev IOPS: 246.086 2022-03-08T00:45:56.899 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Max IOPS: 1482 2022-03-08T00:45:56.899 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Min IOPS: 0 2022-03-08T00:45:56.899 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Average Latency(s): 0.0514083 2022-03-08T00:45:56.899 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Stddev Latency(s): 0.295844 2022-03-08T00:45:56.900 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Max latency(s): 13.0063 2022-03-08T00:45:56.900 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Min latency(s): 0.0015392 2022-03-08T00:45:56.900 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Cleaning up (deleting benchmark objects) 2022-03-08T00:45:56.911 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:56.911513 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:45:56.911 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:56.911538 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:45:57.692 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:45:57.691008 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:46:01.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:01.199177 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:46:04.112 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:04.111306 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:46:06.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:06.682424 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:46:09.396 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:09.395383 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:46:09.399 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:09.397200 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:46:10.375 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:10.374126 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:46:13.524 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:13.524374 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:46:18.653 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:18.649669 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:46:20.870 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:20.870178 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:46:20.944 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:20.943290 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:46:20.944 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:20.943315 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:46:21.598 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:21.595126 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:46:24.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:24.469311 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:46:34.722 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:34.722531 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:46:36.849 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:36.849385 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:46:37.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:37.190581 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:46:37.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:37.190606 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:46:37.778 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:37.773131 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:46:40.631 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:46:40.630282 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:47:01.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:01.122775 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:47:03.317 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:03.316864 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:47:03.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:03.394831 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:47:03.399 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:03.394856 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:47:03.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:03.964042 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:47:06.943 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:06.942821 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:47:48.054 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:48.054234 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:47:50.207 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:50.198376 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:47:50.914 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:50.912597 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:47:50.914 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:50.912623 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:47:51.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:51.601094 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:47:55.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:47:55.090844 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:49:17.021 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:17.020061 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:49:19.129 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:19.129418 I | ceph-object-controller: reconciling object store deployments 2022-03-08T00:49:20.145 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:20.144950 I | ceph-object-controller: ceph object store gateway service running at 10.107.162.74 2022-03-08T00:49:20.146 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:20.144971 I | ceph-object-controller: reconciling object store pools 2022-03-08T00:49:20.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:20.752865 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T00:49:24.104 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:49:24.104344 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T00:51:15.758 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Removed 112148 objects 2022-03-08T00:51:15.759 INFO:tasks.radosbench.radosbench.a.smithi120.stdout:Clean up completed and total clean up time :318.858 2022-03-08T00:51:16.638 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-08T00:51:16.639 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-08T00:51:18.248 INFO:teuthology.orchestra.run.smithi120.stderr:pool 'unique_pool_0' removed 2022-03-08T00:51:18.278 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-08T00:51:18.288 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-08T00:51:18.289 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-08T00:51:18.305 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T00:51:18.316 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:51:19.023 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:51:19.023 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:44:26.000000Z", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:51:18.889707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T00:51:18.889707Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:18.889707Z", "running": 0, "size": 1}}] 2022-03-08T00:51:19.044 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch rm mds.foo 2022-03-08T00:51:19.540 INFO:teuthology.orchestra.run.smithi120.stdout:Removed cephfilesystems/foo 2022-03-08T00:51:19.563 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch rm rbd-mirror 2022-03-08T00:51:19.608 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:19.608092 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:51:20.076 INFO:teuthology.orchestra.run.smithi120.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T00:51:20.101 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch rm rgw.foo 2022-03-08T00:51:20.104 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:20.103960 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T00:51:20.604 INFO:teuthology.orchestra.run.smithi120.stdout:Removed cephobjectstores/foo 2022-03-08T00:51:20.624 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:51:20.945 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:20.892735 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:51:20.946 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:20.935313 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T00:51:21.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:21.783676 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T00:51:21.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:21.783695 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T00:51:21.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:21.783699 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T00:51:24.068 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:51:24.069 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:44:26.000000Z", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:51:21.829120Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:21.829120Z", "running": 0, "size": 1}}] 2022-03-08T00:51:24.480 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:24.443923 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T00:51:25.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:25.011721 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T00:51:28.022 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:28.021699 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T00:51:28.023 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:28.021715 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T00:51:28.023 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:28.021719 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T00:51:29.388 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:29.388542 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T00:51:29.389 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:29.388563 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T00:51:29.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:29.851735 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:51:29.841+0000 7fb442f16340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:51:29.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:51:29.841+0000 7fb442f16340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:51:29.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:51:29.841+0000 7fb442f16340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:51:29.854 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:51:29.854 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:29.851757 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:51:29.841+0000 7fb442f16340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:51:29.854 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:51:29.841+0000 7fb442f16340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:51:29.854 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:51:29.841+0000 7fb442f16340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:51:29.855 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:51:32.477 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:32.477279 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T00:51:34.085 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:51:36.181 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:51:36.182 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:44:26.000000Z", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:51:34.711307Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:34.711307Z", "running": 0, "size": 1}}] 2022-03-08T00:51:37.877 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:37.876525 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T00:51:37.877 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:37.876552 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T00:51:40.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:40.604756 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T00:51:41.020 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:41.020526 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T00:51:41.021 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:41.020555 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T00:51:42.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:42.967807 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T00:51:42.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:42.967849 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T00:51:42.975 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:42.975202 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:51:42.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:42.984724 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:51:43.822 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:43.822441 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T00:51:43.849 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:43.848813 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T00:51:43.858 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:43.858403 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T00:51:43.859 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:43.858567 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T00:51:43.867 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:43.867147 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T00:51:46.208 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:51:47.600 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:51:47.600 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:44:26.000000Z", "last_refresh": "2022-03-08T00:51:47.507016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:51:47.507016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:51:47.507016Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:51:47.507016Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:51:47.507016Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:47.507016Z", "running": 0, "size": 1}}] 2022-03-08T00:51:48.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:51:48.474456 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi120": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T00:51:57.623 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:51:58.277 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:51:58.277 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:44:26.000000Z", "last_refresh": "2022-03-08T00:51:58.147623Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:51:58.147623Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:51:58.147623Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:51:58.147623Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:51:58.147623Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:51:58.147623Z", "running": 0, "size": 1}}] 2022-03-08T00:52:07.956 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:52:07.956249 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T00:52:08.302 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:52:08.604 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:52:08.603389 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:52:08.593+0000 7f2188e65340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:52:08.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:52:08.593+0000 7f2188e65340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:52:08.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:52:08.593+0000 7f2188e65340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:52:08.606 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:52:08.606 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 00:52:08.603420 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:52:08.593+0000 7f2188e65340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:52:08.606 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:52:08.593+0000 7f2188e65340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:52:08.606 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T00:52:08.593+0000 7f2188e65340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:52:08.607 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:52:08.965 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:52:08.966 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:52:08.867618Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:52:08.867618Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:52:08.867618Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:52:08.867618Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:52:08.867618Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:52:08.867618Z", "running": 0, "size": 1}}] 2022-03-08T00:52:18.988 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:52:19.619 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:52:19.620 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:52:19.504132Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:52:19.504132Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:52:19.504132Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:52:19.504132Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:52:19.504132Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:52:19.504132Z", "running": 0, "size": 1}}] 2022-03-08T00:52:29.643 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:52:30.336 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:52:30.337 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:52:30.175326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:52:30.175326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:52:30.175326Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:52:30.175326Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:52:30.175326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:52:30.175326Z", "running": 0, "size": 1}}] 2022-03-08T00:52:40.357 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:52:40.982 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:52:40.983 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:52:40.859271Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:52:40.859271Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:52:40.859271Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:52:40.859271Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:52:40.859271Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:52:40.859271Z", "running": 0, "size": 1}}] 2022-03-08T00:52:51.004 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:52:51.653 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:52:51.653 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:52:51.501167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:52:51.501167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:52:51.501167Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:52:51.501167Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:52:51.501167Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:52:51.501167Z", "running": 0, "size": 1}}] 2022-03-08T00:53:01.679 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:02.292 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:02.358 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:02.199487Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:02.199487Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:02.199487Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:02.199487Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:02.199487Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:02.199487Z", "running": 0, "size": 1}}] 2022-03-08T00:53:12.360 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:13.031 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:13.031 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:12.913952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:12.913952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:12.913952Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:12.913952Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:12.913952Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:12.913952Z", "running": 0, "size": 1}}] 2022-03-08T00:53:23.058 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:23.729 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:23.729 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:23.631077Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:23.631077Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:23.631077Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:23.631077Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:23.631077Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:23.631077Z", "running": 0, "size": 1}}] 2022-03-08T00:53:33.754 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:34.359 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:34.360 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:34.270965Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:34.270965Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:34.270965Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:34.270965Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:34.270965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:34.270965Z", "running": 0, "size": 1}}] 2022-03-08T00:53:44.381 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:44.987 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:44.988 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:44.884473Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:44.884473Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:44.884473Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:44.884473Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:44.884473Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:44.884473Z", "running": 0, "size": 1}}] 2022-03-08T00:53:55.041 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:53:55.676 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:53:55.676 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:53:55.536052Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:53:55.536052Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:53:55.536052Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:53:55.536052Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:53:55.536052Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:53:55.536052Z", "running": 0, "size": 1}}] 2022-03-08T00:54:05.700 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:54:06.323 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:54:06.324 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:54:06.228630Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:54:06.228630Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:54:06.228630Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:54:06.228630Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:54:06.228630Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:54:06.228630Z", "running": 0, "size": 1}}] 2022-03-08T00:54:16.348 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:54:20.020 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:54:20.021 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:54:19.905239Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:54:19.905239Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:54:19.905239Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:54:19.905239Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:54:19.905239Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:54:19.905239Z", "running": 0, "size": 1}}] 2022-03-08T00:54:30.043 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:54:30.713 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:54:30.714 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:54:30.573345Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:54:30.573345Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:54:30.573345Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:54:30.573345Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:54:30.573345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:54:30.573345Z", "running": 0, "size": 1}}] 2022-03-08T00:54:40.735 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:54:41.319 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:54:41.320 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:54:41.212367Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:54:41.212367Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:54:41.212367Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:54:41.212367Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:54:41.212367Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:54:41.212367Z", "running": 0, "size": 1}}] 2022-03-08T00:54:51.344 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:54:52.266 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:54:52.267 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:54:51.871422Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:54:51.871422Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:54:51.871422Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:54:51.871422Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:54:51.871422Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:54:51.871422Z", "running": 0, "size": 1}}] 2022-03-08T00:55:02.286 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:02.875 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:02.875 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:02.786166Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:02.786166Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:02.786166Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:02.786166Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:02.786166Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:02.786166Z", "running": 0, "size": 1}}] 2022-03-08T00:55:12.900 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:13.546 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:13.546 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:13.459887Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:13.459887Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:13.459887Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:13.459887Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:13.459887Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:13.459887Z", "running": 0, "size": 1}}] 2022-03-08T00:55:23.571 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:24.247 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:24.248 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:24.105020Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:24.105020Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:24.105020Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:24.105020Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:24.105020Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:24.105020Z", "running": 0, "size": 1}}] 2022-03-08T00:55:34.272 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:34.883 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:34.884 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:34.792809Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:34.792809Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:34.792809Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:34.792809Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:34.792809Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:34.792809Z", "running": 0, "size": 1}}] 2022-03-08T00:55:44.907 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:45.530 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:45.530 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:45.431783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:45.431783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:45.431783Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:45.431783Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:45.431783Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:45.431783Z", "running": 0, "size": 1}}] 2022-03-08T00:55:55.553 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:55:56.198 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:55:56.199 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:55:56.073913Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:55:56.073913Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:55:56.073913Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:55:56.073913Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:55:56.073913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:55:56.073913Z", "running": 0, "size": 1}}] 2022-03-08T00:56:06.228 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:56:06.925 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:56:06.925 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:56:06.775451Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:56:06.775451Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:56:06.775451Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:56:06.775451Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:56:06.775451Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:56:06.775451Z", "running": 0, "size": 1}}] 2022-03-08T00:56:16.947 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:56:17.579 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:56:17.579 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:56:17.487445Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:56:17.487445Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:56:17.487445Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:56:17.487445Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:56:17.487445Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:56:17.487445Z", "running": 0, "size": 1}}] 2022-03-08T00:56:27.603 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:56:28.175 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:56:28.175 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:56:28.085120Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:56:28.085120Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:56:28.085120Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:56:28.085120Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:56:28.085120Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:56:28.085120Z", "running": 0, "size": 1}}] 2022-03-08T00:56:38.195 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:56:38.823 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:56:38.824 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:56:38.727954Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:56:38.727954Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:56:38.727954Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:56:38.727954Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:56:38.727954Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:56:38.727954Z", "running": 0, "size": 1}}] 2022-03-08T00:56:48.848 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:56:49.491 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:56:49.491 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:56:49.367516Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:56:49.367516Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:56:49.367516Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:56:49.367516Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:56:49.367516Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:56:49.367516Z", "running": 0, "size": 1}}] 2022-03-08T00:56:59.512 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:00.112 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:00.112 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:00.020880Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:00.020880Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:00.020880Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:00.020880Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:00.020880Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:00.020880Z", "running": 0, "size": 1}}] 2022-03-08T00:57:10.131 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:10.771 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:10.771 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:10.670984Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:10.670984Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:10.670984Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:10.670984Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:10.670984Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:10.670984Z", "running": 0, "size": 1}}] 2022-03-08T00:57:20.792 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:21.440 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:21.441 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:21.306371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:21.306371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:21.306371Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:21.306371Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:21.306371Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:21.306371Z", "running": 0, "size": 1}}] 2022-03-08T00:57:31.468 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:32.082 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:32.082 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:31.986873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:31.986873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:31.986873Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:31.986873Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:31.986873Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:31.986873Z", "running": 0, "size": 1}}] 2022-03-08T00:57:42.104 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:42.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:42.748 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:42.645503Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:42.645503Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:42.645503Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:42.645503Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:42.645503Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:42.645503Z", "running": 0, "size": 1}}] 2022-03-08T00:57:52.772 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:57:53.368 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:57:53.369 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:57:53.266870Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:57:53.266870Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:57:53.266870Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:57:53.266870Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:57:53.266870Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:57:53.266870Z", "running": 0, "size": 1}}] 2022-03-08T00:58:03.392 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:04.049 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:04.050 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:03.928205Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:03.928205Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:03.928205Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:03.928205Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:03.928205Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:03.928205Z", "running": 0, "size": 1}}] 2022-03-08T00:58:14.073 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:14.730 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:14.594146Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:14.594146Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:14.594146Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:14.594146Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:14.594146Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:14.594146Z", "running": 0, "size": 1}}] 2022-03-08T00:58:24.755 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:25.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:25.427 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:25.303069Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:25.303069Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:25.303069Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:25.303069Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:25.303069Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:25.303069Z", "running": 0, "size": 1}}] 2022-03-08T00:58:35.445 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:36.074 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:36.074 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:35.966196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:35.966196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:35.966196Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:35.966196Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:35.966196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:35.966196Z", "running": 0, "size": 1}}] 2022-03-08T00:58:46.100 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:46.764 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:46.764 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:46.608112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:46.608112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:46.608112Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:46.608112Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:46.608112Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:46.608112Z", "running": 0, "size": 1}}] 2022-03-08T00:58:56.786 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:58:57.437 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:58:57.437 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:58:57.346368Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:58:57.346368Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:58:57.346368Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:58:57.346368Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:58:57.346368Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:58:57.346368Z", "running": 0, "size": 1}}] 2022-03-08T00:59:07.460 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:59:08.057 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:59:08.057 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:59:07.968876Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:59:07.968876Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:59:07.968876Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:59:07.968876Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:59:07.968876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:59:07.968876Z", "running": 0, "size": 1}}] 2022-03-08T00:59:18.081 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:59:18.784 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:59:18.785 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:59:18.629667Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:59:18.629667Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:59:18.629667Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:59:18.629667Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:59:18.629667Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:59:18.629667Z", "running": 0, "size": 1}}] 2022-03-08T00:59:28.808 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:59:29.404 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:59:29.405 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:59:29.306423Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:59:29.306423Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:59:29.306423Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:59:29.306423Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:59:29.306423Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:59:29.306423Z", "running": 0, "size": 1}}] 2022-03-08T00:59:39.429 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:59:40.055 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:59:40.055 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:59:39.957283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:59:39.957283Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:59:39.957283Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:59:39.957283Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:59:39.957283Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:59:39.957283Z", "running": 0, "size": 1}}] 2022-03-08T00:59:50.078 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T00:59:50.674 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T00:59:50.675 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T00:59:50.586666Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T00:59:50.586666Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T00:59:50.586666Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T00:59:50.586666Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T00:59:50.586666Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T00:59:50.586666Z", "running": 0, "size": 1}}] 2022-03-08T01:00:00.697 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:01.344 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:01.344 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:01.251391Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:01.251391Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:01.251391Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:01.251391Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:01.251391Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:01.251391Z", "running": 0, "size": 1}}] 2022-03-08T01:00:11.369 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:12.001 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:12.001 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:11.901922Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:11.901922Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:11.901922Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:11.901922Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:11.901922Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:11.901922Z", "running": 0, "size": 1}}] 2022-03-08T01:00:22.024 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:22.635 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:22.539309Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:22.539309Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:22.539309Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:22.539309Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:22.539309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:22.539309Z", "running": 0, "size": 1}}] 2022-03-08T01:00:32.659 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:33.366 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:33.366 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:33.206347Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:33.206347Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:33.206347Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:33.206347Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:33.206347Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:33.206347Z", "running": 0, "size": 1}}] 2022-03-08T01:00:43.380 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:44.017 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:44.018 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:43.924357Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:43.924357Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:43.924357Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:43.924357Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:43.924357Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:43.924357Z", "running": 0, "size": 1}}] 2022-03-08T01:00:54.042 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:00:54.722 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:00:54.723 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:00:54.578729Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:00:54.578729Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:00:54.578729Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:00:54.578729Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:00:54.578729Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:00:54.578729Z", "running": 0, "size": 1}}] 2022-03-08T01:01:04.749 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:05.421 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:05.422 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:05.327245Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:05.327245Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:05.327245Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:05.327245Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:05.327245Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:05.327245Z", "running": 0, "size": 1}}] 2022-03-08T01:01:15.443 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:16.080 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:16.080 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:15.987000Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:15.987000Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:15.987000Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:15.987000Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:15.987000Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:15.987000Z", "running": 0, "size": 1}}] 2022-03-08T01:01:26.106 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:26.758 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:26.758 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:26.640315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:26.640315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:26.640315Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:26.640315Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:26.640315Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:26.640315Z", "running": 0, "size": 1}}] 2022-03-08T01:01:36.783 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:37.383 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:37.383 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:37.288966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:37.288966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:37.288966Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:37.288966Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:37.288966Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:37.288966Z", "running": 0, "size": 1}}] 2022-03-08T01:01:47.405 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:48.033 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:48.033 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:47.936982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:47.936982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:47.936982Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:47.936982Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:47.936982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:47.936982Z", "running": 0, "size": 1}}] 2022-03-08T01:01:58.057 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:01:58.676 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:01:58.677 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:01:58.563130Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:01:58.563130Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:01:58.563130Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:01:58.563130Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:01:58.563130Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:01:58.563130Z", "running": 0, "size": 1}}] 2022-03-08T01:02:08.720 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:02:09.315 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:02:09.315 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:02:09.221494Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:02:09.221494Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:02:09.221494Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:02:09.221494Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:02:09.221494Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:02:09.221494Z", "running": 0, "size": 1}}] 2022-03-08T01:02:19.340 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:02:19.921 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:02:19.921 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:02:19.828527Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:02:19.828527Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:02:19.828527Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:02:19.828527Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:02:19.828527Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:02:19.828527Z", "running": 0, "size": 1}}] 2022-03-08T01:02:29.946 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:02:30.627 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:02:30.627 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:02:30.534704Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:02:30.534704Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:02:30.534704Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:02:30.534704Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:02:30.534704Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:02:30.534704Z", "running": 0, "size": 1}}] 2022-03-08T01:02:40.650 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:02:41.281 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:02:41.281 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:02:41.187199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:02:41.187199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:02:41.187199Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:02:41.187199Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:02:41.187199Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:02:41.187199Z", "running": 0, "size": 1}}] 2022-03-08T01:02:51.304 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:02:51.942 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:02:51.943 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:02:51.834362Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:02:51.834362Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:02:51.834362Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:02:51.834362Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:02:51.834362Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:02:51.834362Z", "running": 0, "size": 1}}] 2022-03-08T01:03:01.967 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:02.604 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:02.605 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:02.477029Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:02.477029Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:02.477029Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:02.477029Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:02.477029Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:02.477029Z", "running": 0, "size": 1}}] 2022-03-08T01:03:03.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:03:03.969260 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T01:03:04.576 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:03:04.574321 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:03:04.566+0000 7fe43b9be340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:03:04.577 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:03:04.566+0000 7fe43b9be340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:03:04.578 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:03:04.566+0000 7fe43b9be340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:03:04.579 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:03:04.579 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:03:04.574346 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:03:04.566+0000 7fe43b9be340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:03:04.580 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:03:04.566+0000 7fe43b9be340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:03:04.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:03:04.566+0000 7fe43b9be340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:03:04.581 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:03:12.629 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:13.381 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:13.382 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:13.263315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:13.263315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:13.263315Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:13.263315Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:13.263315Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:13.263315Z", "running": 0, "size": 1}}] 2022-03-08T01:03:23.403 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:23.998 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:23.998 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:23.897323Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:23.897323Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:23.897323Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:23.897323Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:23.897323Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:23.897323Z", "running": 0, "size": 1}}] 2022-03-08T01:03:34.015 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:34.623 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:34.623 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:34.527861Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:34.527861Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:34.527861Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:34.527861Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:34.527861Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:34.527861Z", "running": 0, "size": 1}}] 2022-03-08T01:03:44.649 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:45.289 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:45.289 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:45.188802Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:45.188802Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:45.188802Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:45.188802Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:45.188802Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:45.188802Z", "running": 0, "size": 1}}] 2022-03-08T01:03:55.309 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:03:55.981 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:03:55.982 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:03:55.843185Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:03:55.843185Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:03:55.843185Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:03:55.843185Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:03:55.843185Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:03:55.843185Z", "running": 0, "size": 1}}] 2022-03-08T01:04:06.003 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:04:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:04:06.590 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:04:06.500760Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:04:06.500760Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:04:06.500760Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:04:06.500760Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:04:06.500760Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:04:06.500760Z", "running": 0, "size": 1}}] 2022-03-08T01:04:16.616 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:04:17.433 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:04:17.434 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:04:17.334540Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:04:17.334540Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:04:17.334540Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:04:17.334540Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:04:17.334540Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:04:17.334540Z", "running": 0, "size": 1}}] 2022-03-08T01:04:27.459 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:04:30.859 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:04:30.859 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:04:30.763968Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:04:30.763968Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:04:30.763968Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:04:30.763968Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:04:30.763968Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:04:30.763968Z", "running": 0, "size": 1}}] 2022-03-08T01:04:40.882 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:04:41.468 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:04:41.469 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:04:41.377925Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:04:41.377925Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:04:41.377925Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:04:41.377925Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:04:41.377925Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:04:41.377925Z", "running": 0, "size": 1}}] 2022-03-08T01:04:51.492 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:04:52.133 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:04:52.133 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:04:52.038902Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:04:52.038902Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:04:52.038902Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:04:52.038902Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:04:52.038902Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:04:52.038902Z", "running": 0, "size": 1}}] 2022-03-08T01:05:02.159 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:02.761 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:02.762 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:02.668750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:02.668750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:02.668750Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:02.668750Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:02.668750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:02.668750Z", "running": 0, "size": 1}}] 2022-03-08T01:05:12.785 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:13.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:13.426 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:13.327079Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:13.327079Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:13.327079Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:13.327079Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:13.327079Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:13.327079Z", "running": 0, "size": 1}}] 2022-03-08T01:05:23.452 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:24.118 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:24.118 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:24.016164Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:24.016164Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:24.016164Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:24.016164Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:24.016164Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:24.016164Z", "running": 0, "size": 1}}] 2022-03-08T01:05:34.143 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:34.815 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:34.815 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:34.666008Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:34.666008Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:34.666008Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:34.666008Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:34.666008Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:34.666008Z", "running": 0, "size": 1}}] 2022-03-08T01:05:44.833 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:45.502 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:45.503 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:45.378982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:45.378982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:45.378982Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:45.378982Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:45.378982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:45.378982Z", "running": 0, "size": 1}}] 2022-03-08T01:05:55.526 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:05:56.157 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:05:56.158 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:05:56.069618Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:05:56.069618Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:05:56.069618Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:05:56.069618Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:05:56.069618Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:05:56.069618Z", "running": 0, "size": 1}}] 2022-03-08T01:06:06.182 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:06:06.877 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:06:06.878 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:06:06.691606Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:06:06.691606Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:06:06.691606Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:06:06.691606Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:06:06.691606Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:06:06.691606Z", "running": 0, "size": 1}}] 2022-03-08T01:06:16.900 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:06:17.535 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:06:17.536 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:06:17.442994Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:06:17.442994Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:06:17.442994Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:06:17.442994Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:06:17.442994Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:06:17.442994Z", "running": 0, "size": 1}}] 2022-03-08T01:06:27.558 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:06:28.225 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:06:28.225 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:06:28.106840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:06:28.106840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:06:28.106840Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:06:28.106840Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:06:28.106840Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:06:28.106840Z", "running": 0, "size": 1}}] 2022-03-08T01:06:38.252 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:06:38.908 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:06:38.908 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:06:38.808183Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:06:38.808183Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:06:38.808183Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:06:38.808183Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:06:38.808183Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:06:38.808183Z", "running": 0, "size": 1}}] 2022-03-08T01:06:48.928 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:06:49.529 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:06:49.529 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:06:49.424787Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:06:49.424787Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:06:49.424787Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:06:49.424787Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:06:49.424787Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:06:49.424787Z", "running": 0, "size": 1}}] 2022-03-08T01:06:59.553 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:07:00.215 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:07:00.216 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:07:00.121542Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:07:00.121542Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:07:00.121542Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:07:00.121542Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:07:00.121542Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:07:00.121542Z", "running": 0, "size": 1}}] 2022-03-08T01:07:10.238 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:07:10.836 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:07:10.836 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:07:10.746002Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:07:10.746002Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:07:10.746002Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:07:10.746002Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:07:10.746002Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:07:10.746002Z", "running": 0, "size": 1}}] 2022-03-08T01:07:20.861 DEBUG:teuthology.orchestra.run.smithi120:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-p278g -- ceph orch ls -f json 2022-03-08T01:07:21.480 INFO:teuthology.orchestra.run.smithi120.stdout: 2022-03-08T01:07:21.481 INFO:teuthology.orchestra.run.smithi120.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:51:48.000000Z", "last_refresh": "2022-03-08T01:07:21.387804Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:35:00.000000Z", "last_refresh": "2022-03-08T01:07:21.387804Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:34:35.000000Z", "last_refresh": "2022-03-08T01:07:21.387804Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T00:36:44.000000Z", "last_refresh": "2022-03-08T01:07:21.387804Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T01:07:21.387804Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T01:07:21.387804Z", "running": 0, "size": 1}}] 2022-03-08T01:07:21.502 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T01:07:21.502 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T01:07:21.503 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T01:07:21.503 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T01:07:21.517 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T01:07:21.517 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T01:07:21.577 DEBUG:teuthology.orchestra.remote:smithi120:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T01:07:21.595 DEBUG:teuthology.orchestra.run.smithi120:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T01:07:21.671 INFO:teuthology.orchestra.run.smithi120.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T01:07:21.688 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T01:07:21.688 DEBUG:teuthology.orchestra.run.smithi120:> kubectl delete -f cluster.yaml 2022-03-08T01:07:21.822 INFO:teuthology.orchestra.run.smithi120.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T01:07:21.824 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:21.825048 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T01:07:22.053 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:22.053315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:22.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:22.062343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:22.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:22.062374 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:32.287 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:32.287840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:32.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:32.299672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:42.547 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:42.546428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:42.555 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:42.555522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:52.774 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:52.774377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:07:52.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:07:52.785946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:03.013 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:03.013552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:03.025 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:03.025406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:13.257 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:13.256755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:13.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:13.273094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:23.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:23.503691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:23.553 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:23.553441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:33.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:33.777425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:33.811 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:33.789490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:44.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:44.012456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:44.028 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:44.028085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:54.245 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:54.245567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:08:54.255 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:08:54.255601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:04.481 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:04.481316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:04.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:04.494681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:14.715 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:14.715021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:14.768 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:14.768171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:24.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:24.992544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:25.000 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:25.001060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:35.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:35.247624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:35.261 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:35.261750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:45.484 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:45.483958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:45.498 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:45.498987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:55.715 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:55.715516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:09:55.727 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:09:55.727100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:05.948 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:05.948073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:05.960 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:05.960491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:16.176 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:16.176835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:16.185 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:16.185677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:26.408 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:26.408779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:26.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:26.422262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:36.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:36.642599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:36.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:36.651827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:46.866 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:46.867318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:46.879 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:46.879708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:57.114 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:57.114754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:10:57.127 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:10:57.127683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:07.347 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:07.347589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:07.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:07.361789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:17.618 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:17.618028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:17.626 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:17.626700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:27.846 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:27.846931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:27.858 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:27.857865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:38.078 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:38.078069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:38.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:38.090706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:48.313 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:48.313408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:48.324 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:48.325271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:58.546 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:58.546752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:11:58.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:11:58.558050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:08.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:08.777527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:08.788 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:08.789087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:19.009 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:19.010031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:19.022 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:19.022971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:29.250 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:29.249125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:29.256 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:29.257243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:39.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:39.475866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:39.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:39.487504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:49.735 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:49.736038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:49.745 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:49.746112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:59.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:59.970083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:12:59.985 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:12:59.984394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:10.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:10.211893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:10.224 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:10.224361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:20.497 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:20.497544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:20.507 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:20.508050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:30.726 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:30.726562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:30.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:30.738899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:40.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:40.961700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:40.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:40.973733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:51.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:51.191623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:13:51.200 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:13:51.200182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:01.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:01.434856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:01.448 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:01.449021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:11.666 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:11.667195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:11.681 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:11.681568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:21.923 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:21.923387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:21.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:21.936877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:32.166 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:32.161083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:32.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:32.177984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:42.485 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:42.485702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:42.497 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:42.497675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:52.726 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:52.726464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:14:52.737 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:14:52.738598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:02.960 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:02.961095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:02.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:02.970680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:13.186 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:13.186905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:13.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:13.203019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:23.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:23.423203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:23.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:23.437023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:33.652 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:33.652199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:33.883 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:33.883657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:44.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:44.103808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:44.114 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:44.115133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:54.339 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:54.339768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:15:54.351 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:15:54.351991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:04.607 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:04.607585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:04.618 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:04.619205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:14.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:14.833998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:14.846 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:14.847402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:25.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:25.095438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:25.108 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:25.108715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:35.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:35.327080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:35.338 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:35.338612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:45.558 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:45.558331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:45.571 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:45.572316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:55.807 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:55.807517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:16:55.819 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:16:55.820024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:06.071 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:06.072090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:06.083 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:06.084145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:06.431 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:06.431765 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 01:17:06.393814828 +0000 UTC m=+2634.409785576 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T01:17:16.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:16.299383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:16.310 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:16.311332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:26.528 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:26.528574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:26.538 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:26.539445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:36.759 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:36.760267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:36.772 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:36.773296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:46.993 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:46.993790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:47.003 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:47.003728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:57.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:57.223107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:17:57.235 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:17:57.236304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:07.351 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:07.352350 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 01:18:07.345998359 +0000 UTC m=+2695.361969089 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T01:18:07.454 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:07.455356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:07.468 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:07.468894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:17.688 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:17.687755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:17.698 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:17.698646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:27.958 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:27.958733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:27.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:27.971025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:38.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:38.192122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:38.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:38.203893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:48.484 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:48.484603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:48.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:48.495665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:58.741 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:58.742614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:18:58.752 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:18:58.753093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:09.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:09.010684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:09.021 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:09.022975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:19.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:19.240888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:19.251 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:19.252103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:29.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:29.474720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:29.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:29.488486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:39.713 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:39.713670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:39.722 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:39.723426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:44.583 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:44.584540 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T01:19:45.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:45.198045 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:19:45.189+0000 7f4a8254e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:19:45.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:19:45.189+0000 7f4a8254e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:19:45.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:19:45.189+0000 7f4a8254e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:19:45.198 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:19:45.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:45.198075 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:19:45.189+0000 7f4a8254e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:19:45.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:19:45.189+0000 7f4a8254e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:19:45.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:19:45.189+0000 7f4a8254e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:19:45.202 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:19:49.942 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:49.941821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:19:49.953 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:19:49.953751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:00.171 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:00.172672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:00.183 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:00.184477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:10.406 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:10.407015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:10.419 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:10.420331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:20.640 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:20.637978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:20.650 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:20.650369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:30.875 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:30.874842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:30.893 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:30.894321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:41.115 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:41.115866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:41.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:41.126347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:51.343 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:51.344246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:20:51.354 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:20:51.355477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:01.572 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:01.573301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:01.583 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:01.584876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:11.817 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:11.817644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:11.830 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:11.830128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:22.049 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:22.049965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:22.063 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:22.064208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:32.298 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:32.298946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:32.309 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:32.310255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:42.545 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:42.545741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:42.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:42.558584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:52.776 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:52.777372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:21:52.789 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:21:52.789955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:03.007 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:03.008538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:03.019 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:03.020413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:13.239 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:13.240049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:13.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:13.250435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:23.488 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:23.489164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:23.501 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:23.501327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:33.728 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:33.729271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:33.741 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:33.742601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:43.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:43.966543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:43.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:43.979744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:54.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:54.201003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:22:54.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:22:54.214707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:04.439 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:04.439187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:04.450 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:04.451680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:14.666 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:14.667149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:14.678 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:14.679406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:24.899 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:24.899633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:24.913 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:24.915095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:35.129 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:35.130259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:35.140 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:35.141891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:45.358 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:45.355515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:45.368 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:45.369431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:55.591 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:55.589454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:23:55.600 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:23:55.601622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:05.820 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:05.820808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:05.831 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:05.832490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:16.053 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:16.054068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:16.066 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:16.067190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:26.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:26.287090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:26.297 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:26.298817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:36.520 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:36.521850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:36.533 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:36.534920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:46.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:46.756959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:46.767 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:46.768643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:56.988 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:56.989415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:24:57.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:24:57.002701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:07.223 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:07.224363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:07.233 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:07.234009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:17.452 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:17.453215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:17.462 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:17.462233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:27.677 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:27.677983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:27.690 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:27.691238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:37.924 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:37.925406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:37.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:37.936272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:48.153 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:48.154749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:48.167 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:48.168141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:58.383 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:58.384752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:25:58.393 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:25:58.394109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:08.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:08.617749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:08.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:08.630373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:18.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:18.851257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:18.859 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:18.860561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:29.073 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:29.074143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:29.081 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:29.082573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:39.294 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:39.295352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:39.306 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:39.307362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:49.527 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:49.527699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:49.540 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:49.541861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:59.778 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:59.778704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:26:59.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:26:59.788732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:10.009 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:10.009585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:10.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:10.018553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:20.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:20.237603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:20.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:20.250424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:30.470 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:30.471381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:30.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:30.484402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:40.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:40.695798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:40.704 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:40.705603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:50.928 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:50.929244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:27:50.937 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:27:50.938757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:01.162 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:01.162788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:01.170 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:01.171773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:11.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:11.394745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:11.406 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:11.407383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:21.633 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:21.633745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:21.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:21.647122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:31.869 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:31.870500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:31.885 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:31.884107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:42.107 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:42.108280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:42.122 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:42.123157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:52.345 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:52.345285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:28:52.354 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:28:52.355212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:02.576 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:02.577243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:02.587 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:02.588708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:12.809 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:12.810166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:12.823 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:12.824453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:23.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:23.049170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:23.094 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:23.095409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:33.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:33.329410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:33.339 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:33.340030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:43.608 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:43.608157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:43.619 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:43.619148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:53.867 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:53.867968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:29:53.875 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:29:53.876399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:04.096 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:04.097221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:04.109 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:04.110102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:14.338 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:14.339525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:14.349 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:14.350658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:24.580 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:24.581033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:24.590 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:24.591321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:34.804 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:34.805281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:34.817 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:34.818158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:45.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:45.035755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:45.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:45.048080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:55.284 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:55.285157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:30:55.295 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:30:55.296327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:05.515 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:05.516153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:05.529 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:05.530111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:15.749 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:15.750410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:15.762 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:15.762949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:25.982 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:25.983326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:25.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:25.993528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:36.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:36.216758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:36.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:36.228255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:46.445 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:46.446237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:46.458 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:46.459104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:56.678 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:56.679040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:31:56.691 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:31:56.692400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:06.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:06.913568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:06.929 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:06.930609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:17.149 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:17.150212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:17.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:17.163016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:27.389 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:27.390160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:27.461 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:27.462461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:37.725 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:37.726030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:37.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:37.739422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:47.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:47.962176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:47.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:47.973496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:58.188 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:58.189098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:32:58.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:32:58.201134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:08.423 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:08.424408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:08.460 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:08.461340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:18.723 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:18.723838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:18.736 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:18.737161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:28.958 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:28.958549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:28.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:28.972953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:39.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:39.192540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:39.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:39.204462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:49.426 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:49.427094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:49.435 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:49.435994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:59.657 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:59.658101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:33:59.671 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:33:59.672553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:09.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:09.894923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:09.907 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:09.908207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:20.130 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:20.131244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:20.143 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:20.144349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:30.381 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:30.382275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:30.391 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:30.392581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:40.638 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:40.639207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:40.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:40.652661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:50.895 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:50.896473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:34:50.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:34:50.921950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:01.141 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:01.142152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:01.159 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:01.159844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:11.380 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:11.381499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:11.389 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:11.390663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:21.659 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:21.660432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:21.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:21.673552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:31.892 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:31.893772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:31.905 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:31.906259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:42.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:42.149264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:42.159 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:42.160732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:52.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:52.417482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:35:52.428 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:35:52.429216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:02.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:02.647219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:02.658 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:02.659377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:12.878 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:12.878918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:12.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:12.890051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:23.041 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:23.042974 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 01:36:23.032613805 +0000 UTC m=+3791.048584582 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T01:36:23.104 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:23.106130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:23.117 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:23.118281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:25.205 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:25.206273 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T01:36:25.796 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:25.797407 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:36:25.787+0000 7f3e83299340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:36:25.797 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:36:25.791+0000 7f3e83299340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:36:25.797 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:36:25.791+0000 7f3e83299340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:36:25.797 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:36:25.798 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:25.797434 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:36:25.787+0000 7f3e83299340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:36:25.798 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:36:25.791+0000 7f3e83299340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:36:25.798 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:36:25.791+0000 7f3e83299340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:36:25.798 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:36:33.338 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:33.339041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:33.404 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:33.405548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:43.620 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:43.620526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:43.631 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:43.632525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:53.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:53.851047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:36:53.861 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:36:53.862385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:04.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:04.107738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:04.119 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:04.119687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:14.363 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:14.364213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:14.374 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:14.375470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:24.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:24.594673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:24.606 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:24.607375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:34.826 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:34.827479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:34.837 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:34.839198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:45.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:45.057806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:45.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:45.104866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:55.322 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:55.323503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:37:55.371 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:37:55.372334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:05.607 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:05.608125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:05.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:05.618417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:15.835 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:15.835911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:15.847 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:15.849062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:26.069 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:26.070441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:26.115 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:26.116267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:36.336 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:36.337212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:36.348 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:36.349707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:46.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:46.598025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:46.609 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:46.610797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:56.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:56.830840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:38:56.843 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:38:56.844556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:07.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:07.065379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:07.078 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:07.077995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:17.294 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:17.295367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:17.301 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:17.302710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:27.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:27.523961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:27.533 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:27.534249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:37.758 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:37.759001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:37.769 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:37.770995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:47.991 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:47.992701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:48.005 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:48.006786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:58.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:58.228109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:39:58.239 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:39:58.241121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:08.458 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:08.458815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:08.468 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:08.469717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:18.688 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:18.689403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:18.706 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:18.704970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:28.922 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:28.923091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:28.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:28.937739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:39.157 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:39.158732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:39.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:39.171296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:49.390 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:49.390869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:49.401 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:49.402694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:59.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:59.674298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:40:59.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:40:59.685369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:09.918 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:09.919296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:09.926 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:09.927880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:20.178 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:20.179365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:20.190 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:20.191704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:30.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:30.420372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:30.429 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:30.430926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:40.701 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:40.702128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:40.748 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:40.750029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:50.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:50.968185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:41:50.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:41:50.980362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:01.197 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:01.198706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:01.214 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:01.212229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:11.439 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:11.440595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:11.445 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:11.447456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:21.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:21.666818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:21.678 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:21.678871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:31.897 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:31.897877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:31.908 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:31.909383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:42.128 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:42.129666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:42.141 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:42.142881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:52.363 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:52.364866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:42:52.376 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:42:52.377300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:02.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:02.598964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:02.612 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:02.613973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:12.839 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:12.840323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:12.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:12.852769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:23.072 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:23.073276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:23.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:23.091749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:33.331 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:33.332126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:33.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:33.343177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:43.563 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:43.563259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:43.574 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:43.575945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:53.796 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:53.797562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:43:53.808 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:43:53.810023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:04.030 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:04.031458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:04.041 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:04.042221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:14.262 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:14.262985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:14.274 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:14.275569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:24.497 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:24.497931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:24.511 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:24.508494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:34.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:34.730377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:34.741 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:34.742387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:44.962 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:44.963510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:44.975 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:44.976305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:55.195 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:55.196721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:44:55.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:44:55.207922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:05.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:05.425699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:05.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:05.437620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:15.660 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:15.661689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:15.674 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:15.676070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:25.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:25.901077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:25.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:25.972919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:36.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:36.192699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:36.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:36.290095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:46.507 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:46.508283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:46.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:46.521366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:56.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:56.739587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:45:56.750 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:45:56.751789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:06.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:06.974385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:06.983 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:06.984351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:17.205 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:17.206541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:17.214 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:17.215572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:27.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:27.495222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:27.507 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:27.508396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:37.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:37.757200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:37.766 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:37.767784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:48.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:48.001914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:48.011 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:48.012681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:58.254 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:58.255445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:46:58.266 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:46:58.267669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:08.489 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:08.489210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:08.499 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:08.500307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:18.721 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:18.721966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:18.733 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:18.733619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:28.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:28.971255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:28.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:28.983007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:39.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:39.226795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:39.239 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:39.240964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:49.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:49.464061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:49.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:49.477679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:59.691 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:59.692457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:47:59.703 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:47:59.704417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:09.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:09.969621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:09.982 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:09.982919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:20.196 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:20.197080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:20.205 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:20.205690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:30.452 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:30.453788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:30.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:30.465826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:40.679 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:40.679908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:40.694 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:40.695443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:50.914 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:50.915438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:48:50.927 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:48:50.928316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:01.145 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:01.146258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:01.178 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:01.179716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:11.441 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:11.441295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:11.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:11.454407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:21.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:21.671198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:21.684 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:21.685244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:31.929 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:31.930098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:31.940 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:31.941954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:42.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:42.193981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:42.205 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:42.205539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:52.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:52.454203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:49:52.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:49:52.466751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:02.684 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:02.684980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:02.692 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:02.693212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:12.911 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:12.912654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:12.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:12.997395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:23.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:23.222158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:23.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:23.236814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:33.451 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:33.451507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:33.462 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:33.462508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:43.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:43.684283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:43.697 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:43.697725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:53.919 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:53.920177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:50:53.930 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:50:53.932144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:04.150 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:04.151209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:04.161 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:04.162917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:14.383 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:14.384352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:14.396 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:14.397363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:24.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:24.618599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:24.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:24.630612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:34.855 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:34.856362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:34.866 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:34.868009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:45.088 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:45.088770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:45.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:45.100263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:55.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:55.351145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:51:55.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:51:55.360256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:05.598 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:05.599771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:05.609 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:05.610946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:15.828 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:15.829453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:15.839 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:15.840484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:26.061 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:26.062307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:26.074 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:26.075183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:36.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:36.300385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:36.337 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:36.338980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:46.577 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:46.578472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:46.589 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:46.590259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:56.813 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:56.814764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:52:56.826 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:52:56.826884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:05.809 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:05.810749 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T01:53:06.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:06.416882 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:53:06.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:53:06.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:53:06.417 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:53:06.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:06.416913 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:53:06.418 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:53:06.418 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T01:53:06.406+0000 7f3bc7af1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:53:06.418 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:53:07.051 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:07.051919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:07.063 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:07.064220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:17.277 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:17.278646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:17.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:17.290844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:27.507 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:27.508531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:27.519 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:27.520887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:37.737 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:37.738881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:37.748 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:37.749474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:37.930 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:37.931780 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 01:53:37.922725752 +0000 UTC m=+4825.938696510 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T01:53:47.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:47.970976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:47.988 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:47.989399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:58.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:58.247871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:53:58.260 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:53:58.261993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:08.479 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:08.480503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:08.488 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:08.488900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:18.741 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:18.741159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:18.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:18.754294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:29.000 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:29.001252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:29.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:29.013642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:39.230 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:39.231214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:39.291 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:39.292705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:49.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:49.522373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:49.530 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:49.531752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:59.743 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:59.744866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:54:59.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:54:59.754466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:10.002 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:10.002821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:10.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:10.018685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:20.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:20.238284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:20.251 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:20.251773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:30.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:30.477559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:30.490 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:30.490756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:40.711 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:40.711920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:40.724 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:40.724987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:50.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:50.950919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:55:50.960 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:55:50.961607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:01.202 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:01.203075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:01.210 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:01.211883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:11.459 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:11.460285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:11.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:11.470772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:21.686 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:21.687639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:21.698 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:21.699493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:31.938 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:31.939939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:31.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:31.952271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:42.168 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:42.169683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:42.222 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:42.223421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:52.510 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:52.511740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:56:52.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:56:52.523359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:02.740 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:02.741354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:02.753 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:02.754107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:12.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:12.973446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:12.985 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:12.986446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:23.214 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:23.214992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:23.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:23.228326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:33.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:33.463716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:33.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:33.477450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:43.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:43.695048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:43.703 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:43.704348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:53.923 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:53.923947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:57:53.934 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:57:53.935645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:04.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:04.192665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:04.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:04.202491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:14.440 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:14.441423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:14.451 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:14.452606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:24.705 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:24.706071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:24.717 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:24.719088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:34.961 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:34.962261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:34.983 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:34.984221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:45.202 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:45.203783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:45.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:45.215265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:55.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:55.475649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:58:55.489 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:58:55.490265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:05.707 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:05.708268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:05.719 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:05.721156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:15.939 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:15.940676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:15.952 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:15.953424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:26.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:26.178707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:26.189 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:26.190612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:36.410 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:36.411597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:36.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:36.426165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:46.650 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:46.651478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:46.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:46.664770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:56.885 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:56.885837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T01:59:56.896 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 01:59:56.898238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:07.113 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:07.114216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:07.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:07.126500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:17.369 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:17.371174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:17.381 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:17.382895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:27.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:27.606943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:27.669 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:27.670911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:37.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:37.894738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:37.904 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:37.906304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:48.142 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:48.143049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:48.152 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:48.153943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:58.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:58.454748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:00:58.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:00:58.465148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:08.710 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:08.711413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:08.722 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:08.723746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:18.960 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:18.960929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:18.989 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:18.991072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:29.214 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:29.214965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:29.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:29.227159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:39.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:39.475703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:39.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:39.488506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:49.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:49.733814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:49.741 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:49.742727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:59.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:59.971515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:01:59.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:01:59.983103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:10.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:10.201193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:10.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:10.213012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:20.433 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:20.434929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:20.447 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:20.448336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:30.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:30.674586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:30.684 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:30.686025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:40.938 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:40.939755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:40.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:40.952901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:51.209 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:51.210337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:02:51.222 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:02:51.223505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:01.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:01.484526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:01.495 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:01.497292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:11.716 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:11.717566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:11.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:11.729832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:21.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:21.952269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:21.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:21.965364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:32.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:32.183712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:32.195 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:32.196590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:42.414 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:42.415527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:42.427 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:42.428857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:52.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:52.646972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:03:52.653 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:03:52.655152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:02.876 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:02.876539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:02.897 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:02.898234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:13.119 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:13.120320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:13.142 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:13.134140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:23.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:23.360764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:23.372 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:23.373356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:33.595 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:33.597235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:33.607 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:33.608058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:43.844 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:43.845626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:43.872 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:43.874340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:54.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:54.091686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:04:54.101 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:04:54.102440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:04.324 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:04.325280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:04.336 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:04.337941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:14.553 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:14.554707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:14.565 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:14.567277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:24.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:24.789077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:24.801 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:24.802528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:35.030 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:35.031293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:35.040 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:35.041782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:45.257 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:45.258192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:45.269 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:45.270682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:55.512 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:55.512636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:05:55.524 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:05:55.526067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:05.740 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:05.741980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:05.748 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:05.750427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:15.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:15.968343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:15.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:15.980494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:26.196 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:26.198276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:26.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:26.207344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:36.425 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:36.426278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:36.437 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:36.438639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:46.659 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:46.660566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:46.671 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:46.672781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:56.943 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:56.944799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:06:56.955 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:06:56.956739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:07.174 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:07.176019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:07.186 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:07.188016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:17.411 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:17.411889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:17.484 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:17.485788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:27.705 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:27.706320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:27.718 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:27.719535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:27.718 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:27.719582 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:37.941 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:37.942178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:37.953 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:37.955371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:48.172 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:48.173977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:48.186 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:48.188059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:58.409 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:58.410660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:07:58.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:07:58.423687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:08.644 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:08.645250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:08.656 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:08.657950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:18.873 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:18.875110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:18.883 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:18.884466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:29.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:29.104653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:29.112 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:29.113761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:39.337 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:39.338133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:39.349 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:39.351071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:49.571 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:49.573145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:49.583 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:49.584592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:59.799 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:59.800919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:08:59.809 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:08:59.811714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:10.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:10.032491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:10.044 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:10.046180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:20.268 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:20.269541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:20.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:20.281654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:30.534 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:30.535464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:30.545 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:30.546652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:40.771 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:40.772897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:40.823 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:40.824846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:46.428 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:46.429389 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T02:09:47.034 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:47.035350 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:09:47.024+0000 7fbdd851f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:09:47.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:09:47.024+0000 7fbdd851f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:09:47.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:09:47.024+0000 7fbdd851f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:09:47.035 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:09:47.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:47.035383 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:09:47.024+0000 7fbdd851f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:09:47.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:09:47.024+0000 7fbdd851f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:09:47.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:09:47.024+0000 7fbdd851f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:09:47.036 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:09:51.062 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:51.063709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:09:51.071 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:09:51.073348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:01.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:01.290924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:01.302 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:01.303876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:11.522 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:11.523697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:11.536 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:11.536359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:21.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:21.755932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:21.767 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:21.768569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:31.986 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:31.988128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:32.002 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:32.004380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:42.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:42.267996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:42.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:42.281045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:52.495 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:52.496010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:52.504 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:52.506570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:10:52.819 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:10:52.821395 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 02:10:52.811564195 +0000 UTC m=+5860.827534981 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T02:11:02.728 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:02.729938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:02.743 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:02.744780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:12.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:12.968033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:12.980 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:12.981296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:23.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:23.200861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:23.212 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:23.213639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:33.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:33.438209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:33.448 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:33.450588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:43.677 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:43.679127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:43.688 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:43.690684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:53.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:53.852757 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 02:11:53.72389355 +0000 UTC m=+5921.739864241 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T02:11:53.925 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:53.927318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:11:53.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:11:53.938491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:04.217 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:04.218522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:04.233 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:04.234383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:14.448 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:14.450407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:14.459 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:14.461498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:24.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:24.682692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:25.049 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:24.695512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:34.917 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:34.919231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:34.930 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:34.932624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:45.150 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:45.151565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:45.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:45.203228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:55.425 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:55.426038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:12:55.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:12:55.440314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:05.657 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:05.659422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:05.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:05.672130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:15.891 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:15.892682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:15.905 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:15.906563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:26.124 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:26.126009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:26.137 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:26.138587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:36.393 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:36.394707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:36.404 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:36.406456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:46.668 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:46.670247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:46.680 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:46.682546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:56.908 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:56.909939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:13:56.918 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:13:56.920401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:07.137 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:07.137399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:07.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:07.150133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:17.371 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:17.372871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:17.384 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:17.385889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:27.607 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:27.608186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:27.618 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:27.620032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:37.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:37.854663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:37.863 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:37.865524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:48.082 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:48.083917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:48.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:48.094621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:58.314 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:58.316399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:14:58.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:14:58.328168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:08.549 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:08.550767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:08.563 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:08.565765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:18.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:18.785869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:18.796 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:18.797401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:29.018 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:29.020004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:29.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:29.033155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:39.256 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:39.258359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:39.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:39.268981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:49.495 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:49.496881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:49.508 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:49.509625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:59.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:59.734055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:15:59.746 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:15:59.747624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:09.967 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:09.968570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:09.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:09.982868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:20.243 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:20.243938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:20.254 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:20.256458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:30.482 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:30.483995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:30.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:30.493752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:40.739 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:40.741589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:40.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:40.756114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:50.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:50.974985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:16:50.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:16:50.985188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:01.204 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:01.206692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:01.217 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:01.219246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:11.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:11.435943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:11.448 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:11.450400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:21.673 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:21.674806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:21.685 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:21.687029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:31.941 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:31.940988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:31.949 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:31.951232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:42.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:42.208187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:42.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:42.218660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:52.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:52.439218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:17:52.579 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:17:52.453393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:02.671 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:02.673082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:02.687 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:02.689712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:12.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:12.966366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:12.974 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:12.976201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:23.194 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:23.195613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:23.205 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:23.206799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:33.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:33.456374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:33.466 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:33.468218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:43.685 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:43.686684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:43.697 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:43.698928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:53.928 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:53.930260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:18:53.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:18:53.952952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:04.183 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:04.185116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:04.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:04.194358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:14.412 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:14.414029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:14.441 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:14.442651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:24.663 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:24.665071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:24.676 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:24.677963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:34.898 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:34.899819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:34.907 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:34.909105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:45.123 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:45.125552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:45.134 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:45.136673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:55.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:55.362634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:19:55.375 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:19:55.377220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:05.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:05.599684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:05.610 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:05.612297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:15.837 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:15.838964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:15.850 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:15.851660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:26.072 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:26.074046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:26.084 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:26.086489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:36.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:36.328409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:36.335 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:36.337271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:46.575 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:46.576938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:46.587 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:46.589457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:56.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:56.811774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:20:56.821 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:20:56.822644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:07.062 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:07.063522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:07.074 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:07.076614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:17.295 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:17.296651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:17.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:17.310354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:27.529 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:27.530431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:27.542 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:27.544314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:37.765 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:37.766791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:37.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:37.779067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:47.997 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:47.999276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:48.008 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:48.010300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:58.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:58.229293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:21:58.240 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:21:58.241841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:08.461 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:08.463570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:08.470 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:08.472297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:18.690 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:18.691511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:18.700 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:18.702672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:28.920 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:28.922190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:28.938 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:28.937039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:39.175 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:39.175399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:39.185 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:39.186261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:49.421 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:49.423608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:49.432 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:49.434340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:59.650 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:59.652227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:22:59.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:22:59.665967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:09.901 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:09.903122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:09.911 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:09.913428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:20.144 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:20.145401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:20.155 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:20.156792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:30.378 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:30.379653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:30.392 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:30.394270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:40.608 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:40.610297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:40.619 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:40.619986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:50.838 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:50.839199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:23:50.879 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:23:50.874736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:01.117 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:01.118951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:01.132 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:01.132966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:11.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:11.360389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:11.370 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:11.372233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:21.590 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:21.591665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:21.602 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:21.603534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:31.821 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:31.822965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:31.831 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:31.833088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:42.050 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:42.050707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:42.061 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:42.062789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:52.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:52.283764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:24:52.294 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:24:52.296024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:02.515 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:02.517198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:02.528 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:02.529807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:12.827 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:12.828548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:12.838 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:12.839906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:23.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:23.065593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:23.073 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:23.075134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:33.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:33.342814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:33.354 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:33.355641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:43.585 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:43.586379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:43.641 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:43.641616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:53.859 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:53.861234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:25:53.872 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:25:53.874170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:04.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:04.243440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:04.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:04.254087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:14.482 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:14.483728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:14.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:14.495636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:24.713 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:24.714894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:24.724 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:24.725957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:27.043 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:27.044551 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T02:26:27.653 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:27.654026 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:26:27.646+0000 7f12117eb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:26:27.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:26:27.646+0000 7f12117eb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:26:27.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:26:27.646+0000 7f12117eb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:26:27.654 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:26:27.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:27.654055 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:26:27.646+0000 7f12117eb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:26:27.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:26:27.646+0000 7f12117eb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:26:27.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:26:27.646+0000 7f12117eb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:26:27.656 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:26:35.191 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:35.192981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:35.207 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:35.208998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:45.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:45.439822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:45.490 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:45.491728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:55.710 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:55.712139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:26:55.723 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:26:55.724536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:05.964 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:05.966057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:05.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:05.975016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:16.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:16.200514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:16.207 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:16.209402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:26.460 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:26.461631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:26.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:26.471289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:36.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:36.739685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:36.749 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:36.750975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:47.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:47.003014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:47.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:47.011505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:57.231 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:57.231806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:27:57.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:27:57.244347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:07.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:07.463848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:07.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:07.476691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:17.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:17.694351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:17.705 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:17.706776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:27.925 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:27.926770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:27.935 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:27.937109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:38.158 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:38.159047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:38.171 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:38.172584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:48.404 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:48.405751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:48.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:48.418984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:58.661 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:58.662613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:28:58.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:28:58.671600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:08.803 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:08.805250 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 02:29:08.777322589 +0000 UTC m=+6956.793293301 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T02:29:08.986 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:08.987585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:08.999 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:09.000842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:19.223 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:19.223824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:19.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:19.236027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:29.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:29.454150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:29.467 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:29.468847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:39.788 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:39.789306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:39.801 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:39.802512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:50.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:50.078398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:29:50.085 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:29:50.086824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:00.300 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:00.300472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:00.357 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:00.357366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:10.630 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:10.631137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:10.641 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:10.643197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:20.890 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:20.891174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:20.900 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:20.902055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:31.118 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:31.119432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:31.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:31.127495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:41.378 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:41.379965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:41.392 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:41.394030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:51.610 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:51.611625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:30:51.621 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:30:51.622799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:01.870 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:01.871742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:01.911 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:01.885165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:12.101 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:12.102715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:12.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:12.146616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:22.367 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:22.367668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:22.399 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:22.378105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:32.600 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:32.600797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:32.614 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:32.614960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:42.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:42.834181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:42.846 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:42.847202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:53.066 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:53.066881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:31:53.076 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:31:53.077608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:03.296 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:03.296739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:03.311 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:03.311469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:13.527 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:13.527561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:13.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:13.540236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:23.757 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:23.758606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:23.766 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:23.767843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:33.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:33.985666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:33.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:33.997859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:44.244 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:44.245603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:44.258 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:44.258685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:54.520 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:54.521645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:32:54.532 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:32:54.533777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:04.751 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:04.752128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:04.926 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:04.764817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:15.230 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:15.231666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:15.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:15.243700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:25.473 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:25.473832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:25.485 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:25.486309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:35.705 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:35.706342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:35.717 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:35.717484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:46.204 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:46.205347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:46.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:46.214836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:56.477 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:56.477817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:33:56.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:33:56.490888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:06.710 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:06.711347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:06.723 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:06.724476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:17.290 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:17.290904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:17.301 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:17.302132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:27.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:27.522540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:27.535 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:27.536808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:37.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:37.756622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:37.771 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:37.771935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:48.009 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:48.009412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:48.020 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:48.021943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:58.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:58.243267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:34:58.254 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:34:58.255912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:08.517 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:08.516950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:08.530 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:08.530282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:18.749 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:18.750105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:18.761 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:18.762015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:28.983 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:28.984598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:28.994 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:28.995430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:39.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:39.222413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:39.234 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:39.235203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:49.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:49.492041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:49.502 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:49.502754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:59.722 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:59.723548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:35:59.737 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:35:59.738113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:09.999 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:10.000233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:10.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:10.018733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:20.269 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:20.270678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:20.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:20.283212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:30.500 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:30.501748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:30.512 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:30.514278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:40.734 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:40.735550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:40.748 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:40.748810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:50.994 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:50.995804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:36:51.006 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:36:51.007482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:01.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:01.225925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:01.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:01.237782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:11.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:11.465869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:11.477 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:11.478232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:21.776 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:21.776952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:21.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:21.788666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:32.025 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:32.025011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:32.034 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:32.034018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:42.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:42.268529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:42.283 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:42.284554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:52.587 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:52.588498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:37:52.938 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:37:52.939642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:03.165 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:03.161684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:03.174 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:03.175211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:13.391 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:13.391880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:13.411 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:13.403097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:23.759 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:23.760408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:23.769 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:23.770712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:34.006 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:34.007119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:34.018 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:34.019218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:44.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:44.247810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:44.257 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:44.258547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:54.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:54.488779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:38:54.497 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:38:54.498430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:04.737 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:04.729364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:04.773 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:04.774797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:14.998 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:14.999765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:15.011 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:15.011930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:25.387 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:25.388503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:25.397 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:25.397390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:35.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:35.618106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:35.626 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:35.627677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:45.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:45.858183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:45.869 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:45.870488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:56.367 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:56.368234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:39:56.380 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:39:56.381401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:06.602 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:06.602890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:06.614 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:06.614736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:16.831 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:16.832214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:16.844 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:16.845674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:27.185 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:27.185690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:27.196 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:27.196109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:37.451 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:37.452235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:37.460 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:37.460811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:47.709 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:47.709964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:47.723 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:47.724227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:58.355 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:58.356001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:40:58.366 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:40:58.367671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:08.580 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:08.580965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:08.592 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:08.592925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:18.873 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:18.873153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:18.879 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:18.880756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:19.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:19.362150 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 02:41:19.355727752 +0000 UTC m=+7687.371698479 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T02:41:29.315 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:29.315824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:29.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:29.326549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:39.547 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:39.547539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:39.559 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:39.560272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:49.782 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:49.782788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:41:49.794 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:41:49.795412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:00.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:00.266284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:00.544 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:00.544623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:10.769 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:10.769792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:10.780 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:10.781109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:21.028 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:21.028406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:21.037 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:21.038353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:31.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:31.425473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:31.433 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:31.434790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:41.650 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:41.651375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:41.662 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:41.663288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:51.880 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:51.881206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:42:51.891 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:42:51.892286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:02.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:02.469814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:02.480 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:02.480729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:07.660 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:07.660678 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T02:43:08.266 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:08.266500 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:43:08.259+0000 7ff74ac39340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:43:08.266 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:43:08.259+0000 7ff74ac39340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:43:08.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:43:08.259+0000 7ff74ac39340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:43:08.267 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:43:08.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:08.266525 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:43:08.259+0000 7ff74ac39340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:43:08.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:43:08.259+0000 7ff74ac39340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:43:08.268 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:43:08.259+0000 7ff74ac39340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:43:08.268 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:43:12.696 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:12.697327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:12.708 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:12.708816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:22.925 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:22.926216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:22.939 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:22.939795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:33.275 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:33.275646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:33.616 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:33.616990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:43.839 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:43.839453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:43.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:43.852271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:54.064 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:54.064869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:43:54.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:43:54.077844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:04.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:04.328574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:04.347 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:04.347827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:14.566 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:14.566268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:14.577 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:14.578447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:24.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:24.833693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:24.841 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:24.842133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:35.068 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:35.068345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:35.080 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:35.080819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:45.306 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:45.306179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:45.317 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:45.318380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:55.535 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:55.535884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:44:55.549 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:44:55.550016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:05.764 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:05.764418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:05.776 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:05.776624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:15.997 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:15.997202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:16.009 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:16.009568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:26.228 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:26.228476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:26.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:26.239122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:36.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:36.455753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:36.471 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:36.471910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:46.754 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:46.754705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:46.769 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:46.770202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:56.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:56.985299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:45:56.995 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:45:56.996174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:07.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:07.216893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:07.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:07.226038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:17.443 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:17.444182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:17.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:17.457020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:27.677 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:27.677705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:27.686 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:27.686651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:37.904 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:37.904132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:37.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:37.912797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:48.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:48.169474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:48.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:48.183301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:58.543 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:58.543538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:46:58.555 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:46:58.555644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:08.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:08.777453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:08.789 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:08.788731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:19.007 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:19.007321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:19.018 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:19.018975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:29.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:29.238800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:29.252 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:29.252215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:39.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:39.475445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:39.488 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:39.488376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:49.717 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:49.717755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:49.728 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:49.728683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:47:59.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:47:59.991329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:00.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:00.017070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:10.241 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:10.234155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:10.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:10.247214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:20.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:20.465879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:20.480 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:20.479156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:30.698 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:30.698109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:30.711 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:30.711908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:40.931 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:40.931872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:40.944 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:40.944865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:51.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:51.169485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:48:51.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:48:51.177960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:01.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:01.398843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:01.412 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:01.413079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:11.662 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:11.662954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:11.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:11.673387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:21.892 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:21.892025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:21.904 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:21.904579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:32.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:32.125933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:32.136 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:32.136829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:42.360 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:42.360052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:42.371 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:42.371499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:52.591 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:52.591881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:49:52.604 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:49:52.604794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:02.864 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:02.864242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:02.878 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:02.878390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:13.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:13.098954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:13.143 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:13.143692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:23.376 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:23.376566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:23.387 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:23.387858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:33.608 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:33.608278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:33.617 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:33.617998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:43.832 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:43.832938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:43.847 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:43.842712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:54.060 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:54.059478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:50:54.082 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:50:54.072698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:04.287 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:04.287709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:04.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:04.299276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:14.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:14.521144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:14.533 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:14.533674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:24.775 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:24.774332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:24.786 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:24.786618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:35.006 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:35.006598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:35.015 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:35.016087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:45.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:45.238221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:45.246 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:45.246510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:55.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:55.466099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:51:55.475 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:51:55.475846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:05.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:05.695331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:05.708 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:05.708781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:15.942 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:15.942049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:15.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:15.952181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:26.170 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:26.170687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:26.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:26.178341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:36.400 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:36.400250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:36.414 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:36.415353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:46.631 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:46.631923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:46.643 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:46.643578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:56.862 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:56.862614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:52:56.873 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:52:56.873179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:07.088 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:07.088661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:07.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:07.098506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:17.348 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:17.349076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:17.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:17.359950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:27.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:27.581719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:27.589 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:27.589954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:37.806 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:37.806775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:37.821 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:37.821273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:48.044 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:48.044281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:48.055 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:48.055924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:58.277 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:58.277702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:53:58.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:53:58.288971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:08.511 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:08.512072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:08.523 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:08.522667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:18.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:18.743940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:18.759 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:18.754735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:28.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:28.982283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:28.993 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:28.993692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:39.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:39.236115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:39.248 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:39.248645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:49.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:49.465774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:49.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:49.483531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:59.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:59.732414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:54:59.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:54:59.744980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:09.963 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:09.963512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:09.977 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:09.977529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:20.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:20.221359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:20.233 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:20.232858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:30.452 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:30.452304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:30.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:30.465403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:40.687 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:40.687092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:40.699 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:40.699305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:50.923 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:50.923735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:55:50.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:55:50.936999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:01.157 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:01.157498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:01.171 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:01.172059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:11.391 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:11.390952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:11.402 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:11.403061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:21.621 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:21.622157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:21.634 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:21.632943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:31.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:31.853452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:31.863 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:31.863590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:42.084 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:42.084003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:42.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:42.095630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:52.348 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:52.348238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:56:52.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:56:52.359801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:02.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:02.581530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:02.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:02.593574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:12.813 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:12.813474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:12.824 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:12.825172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:23.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:23.048359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:23.059 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:23.059601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:33.275 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:33.275525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:33.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:33.285345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:33.410 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:33.410021 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 02:57:33.400539586 +0000 UTC m=+8661.416510337 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T02:57:43.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:43.504973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:43.516 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:43.516963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:53.739 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:53.739064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:57:53.753 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:57:53.752564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:03.973 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:03.973258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:03.990 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:03.990131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:14.209 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:14.209452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:14.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:14.221433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:24.441 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:24.441259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:24.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:24.453462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:34.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:34.671699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:34.685 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:34.685685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:44.908 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:44.908860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:44.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:44.921746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:55.141 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:55.141139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:58:55.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:58:55.182584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:05.426 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:05.426658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:05.439 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:05.439876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:15.682 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:15.681971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:15.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:15.695421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:25.916 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:25.916109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:25.927 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:25.926548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:36.144 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:36.144180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:36.156 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:36.156813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:46.381 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:46.381455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:46.395 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:46.395080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:48.274 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:48.274585 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T02:59:48.920 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:48.916135 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:59:48.905+0000 7fd16e455340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:59:48.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:59:48.905+0000 7fd16e455340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:59:48.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:59:48.905+0000 7fd16e455340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:59:48.921 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:59:48.921 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:48.916235 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:59:48.905+0000 7fd16e455340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:59:48.922 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:59:48.905+0000 7fd16e455340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:59:48.922 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T02:59:48.905+0000 7fd16e455340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:59:48.922 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:59:56.611 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:56.611408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T02:59:56.625 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 02:59:56.625710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:06.844 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:06.844458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:06.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:06.856379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:17.074 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:17.074591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:17.085 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:17.085800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:27.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:27.308620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:27.320 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:27.320861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:37.536 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:37.535307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:37.547 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:37.547781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:47.765 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:47.765437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:47.774 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:47.774539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:57.994 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:57.994298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:00:58.005 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:00:58.005245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:08.230 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:08.230571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:08.244 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:08.244414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:18.468 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:18.467884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:18.481 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:18.481525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:28.707 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:28.706979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:28.719 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:28.719891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:38.940 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:38.940093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:38.951 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:38.951678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:49.176 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:49.175894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:49.208 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:49.208068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:59.436 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:59.436997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:01:59.449 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:01:59.450246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:09.679 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:09.679192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:09.690 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:09.690133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:19.953 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:19.952893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:19.962 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:19.962470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:30.181 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:30.181647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:30.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:30.192537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:40.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:40.416653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:40.426 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:40.426378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:50.642 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:50.641969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:02:50.658 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:02:50.658969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:00.878 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:00.878115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:00.891 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:00.891568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:11.111 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:11.111517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:11.125 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:11.125637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:21.344 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:21.343810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:21.405 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:21.405417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:31.627 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:31.626859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:31.639 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:31.639546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:41.862 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:41.862331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:41.876 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:41.876534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:52.132 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:52.131564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:03:52.140 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:03:52.140985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:02.366 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:02.365442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:02.386 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:02.379997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:12.597 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:12.597337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:12.611 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:12.610858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:22.860 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:22.860129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:22.872 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:22.872663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:33.134 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:33.134343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:33.144 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:33.144722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:43.374 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:43.373738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:43.385 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:43.385611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:53.610 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:53.610294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:04:53.622 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:04:53.622386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:03.861 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:03.861661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:03.871 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:03.871737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:14.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:14.095405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:14.111 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:14.109670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:24.334 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:24.334135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:24.345 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:24.345714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:34.567 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:34.567395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:34.578 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:34.578631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:44.799 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:44.798749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:44.814 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:44.814483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:55.073 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:55.070485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:05:55.083 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:05:55.083186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:05.304 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:05.304300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:05.316 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:05.316467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:15.540 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:15.539706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:15.552 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:15.552499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:25.815 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:25.815580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:25.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:25.829673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:36.070 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:36.070336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:36.078 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:36.079065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:46.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:46.340643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:46.353 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:46.352898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:56.573 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:56.573140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:06:56.584 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:06:56.584634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:06.805 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:06.804912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:06.816 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:06.816633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:17.051 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:17.049976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:17.062 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:17.062982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:27.287 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:27.287777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:27.314 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:27.314888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:37.570 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:37.570688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:37.585 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:37.584976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:37.585 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:37.585010 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:47.802 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:47.802267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:47.817 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:47.817397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:58.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:58.034915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:07:58.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:07:58.047393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:08.281 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:08.281569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:08.293 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:08.294051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:18.530 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:18.530428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:18.542 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:18.542664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:28.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:28.787320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:28.799 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:28.799382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:39.024 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:39.024454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:39.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:39.036125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:49.256 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:49.256376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:49.268 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:49.268279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:59.489 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:59.488988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:08:59.503 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:08:59.502128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:09.721 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:09.721184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:09.733 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:09.733335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:19.955 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:19.955145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:19.971 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:19.969692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:30.189 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:30.189716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:30.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:30.203049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:40.425 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:40.424246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:40.439 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:40.439345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:50.658 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:50.658141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:09:50.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:09:50.669903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:00.892 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:00.891238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:00.905 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:00.905138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:11.129 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:11.129635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:11.143 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:11.143285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:21.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:21.361848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:21.375 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:21.374806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:31.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:31.604503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:31.620 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:31.620992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:41.882 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:41.881855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:41.895 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:41.891872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:52.112 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:52.111742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:10:52.142 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:10:52.124487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:02.342 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:02.341696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:02.356 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:02.355643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:12.578 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:12.577882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:12.590 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:12.590532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:22.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:22.809531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:22.823 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:22.822453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:33.043 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:33.042357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:33.054 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:33.054197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:43.274 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:43.273478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:43.284 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:43.283932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:53.503 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:53.503307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:11:53.516 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:11:53.515687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:03.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:03.732285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:03.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:03.744643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:13.963 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:13.962932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:13.977 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:13.976808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:24.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:24.198786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:24.212 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:24.210931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:34.449 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:34.448678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:34.462 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:34.461796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:44.682 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:44.682101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:44.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:44.693497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:54.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:54.911670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:12:54.924 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:12:54.924189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:05.145 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:05.144353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:05.158 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:05.157391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:15.381 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:15.380640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:15.394 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:15.393312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:25.618 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:25.617536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:25.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:25.628989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:35.849 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:35.848711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:35.861 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:35.860272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:46.084 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:46.082932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:46.099 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:46.096853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:56.318 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:56.317194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:13:56.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:13:56.326142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:06.553 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:06.551962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:06.564 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:06.563923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:16.787 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:16.785728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:16.800 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:16.799595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:27.021 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:27.020143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:27.032 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:27.031356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:37.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:37.287997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:37.297 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:37.296606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:47.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:47.519929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:47.529 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:47.528593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:57.767 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:57.766184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:14:57.779 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:14:57.778383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:07.997 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:07.996714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:08.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:08.008939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:18.228 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:18.227625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:18.242 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:18.241168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:28.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:28.462283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:28.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:28.475149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:38.696 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:38.694763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:38.707 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:38.705372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:48.925 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:48.924032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:48.935 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:48.934640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:59.160 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:59.158501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:15:59.172 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:15:59.171234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:09.388 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:09.387150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:09.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:09.397146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:19.616 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:19.615493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:19.627 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:19.626367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:28.958 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:28.956688 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T03:16:29.559 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:29.557079 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:16:29.547+0000 7f3202593340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:16:29.559 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:16:29.547+0000 7f3202593340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:16:29.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:16:29.547+0000 7f3202593340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:16:29.560 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:16:29.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:29.557106 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:16:29.547+0000 7f3202593340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:16:29.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:16:29.547+0000 7f3202593340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:16:29.561 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:16:29.547+0000 7f3202593340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:16:29.561 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:16:29.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:29.893176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:29.905 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:29.904504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:40.124 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:40.122688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:40.132 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:40.131245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:50.346 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:50.345085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:50.357 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:50.356254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:16:50.413 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:16:50.412598 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 03:16:50.350092656 +0000 UTC m=+9818.366063355 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T03:17:00.586 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:00.585449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:00.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:00.593696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:10.815 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:10.814317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:10.828 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:10.827103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:21.046 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:21.044779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:21.059 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:21.058395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:31.281 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:31.278909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:31.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:31.288344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:41.511 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:41.509307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:41.523 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:41.521269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:51.743 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:51.741771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:17:51.755 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:17:51.753854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:01.976 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:01.975151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:01.993 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:01.991884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:12.214 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:12.212918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:12.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:12.224581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:22.447 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:22.445733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:22.457 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:22.455911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:32.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:32.691597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:32.704 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:32.703078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:42.933 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:42.931876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:42.945 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:42.943912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:53.243 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:53.241218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:18:53.254 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:18:53.253077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:03.476 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:03.474465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:03.487 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:03.486032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:13.728 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:13.726809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:13.751 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:13.749406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:24.045 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:24.043097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:24.054 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:24.052853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:34.418 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:34.416606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:34.430 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:34.429060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:44.652 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:44.651041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:44.662 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:44.661034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:55.015 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:55.014172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:19:55.025 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:19:55.023634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:05.292 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:05.290687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:05.303 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:05.301635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:15.520 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:15.517995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:15.537 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:15.535461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:25.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:25.754998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:25.768 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:25.766251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:35.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:35.990952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:36.005 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:36.004157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:46.227 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:46.226191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:46.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:46.237100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:56.457 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:56.455511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:20:56.470 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:20:56.468400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:06.697 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:06.694359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:06.709 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:06.707637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:16.931 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:16.929472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:16.944 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:16.941684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:27.167 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:27.165464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:27.178 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:27.175948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:37.422 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:37.420640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:37.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:37.432967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:47.668 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:47.666736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:47.679 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:47.677514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:57.903 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:57.901302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:21:57.915 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:21:57.913735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:08.160 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:08.157692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:08.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:08.174876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:18.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:18.414451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:18.427 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:18.425384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:28.676 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:28.674209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:28.689 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:28.687089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:38.901 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:38.899729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:38.909 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:38.907975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:49.126 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:49.124107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:49.140 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:49.137657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:59.371 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:59.369308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:22:59.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:22:59.413640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:09.637 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:09.635350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:09.649 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:09.648007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:19.870 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:19.868204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:19.881 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:19.879706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:30.098 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:30.096317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:30.113 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:30.110988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:40.337 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:40.334876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:40.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:40.348096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:50.568 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:50.566228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:23:50.630 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:23:50.628817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:00.881 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:00.879119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:00.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:00.891764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:11.138 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:11.136538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:11.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:11.146707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:21.368 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:21.366030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:21.381 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:21.379130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:31.610 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:31.607657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:31.623 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:31.620771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:41.841 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:41.838920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:41.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:41.851345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:52.070 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:52.068012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:24:52.084 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:24:52.081705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:02.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:02.296854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:02.310 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:02.308298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:12.527 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:12.525404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:12.539 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:12.537590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:22.758 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:22.755958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:22.770 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:22.768345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:32.989 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:32.986569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:33.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:32.998860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:43.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:43.218834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:43.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:43.234465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:53.516 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:53.513689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:25:53.529 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:25:53.526870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:03.763 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:03.761561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:03.772 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:03.769979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:13.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:13.990294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:14.004 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:14.002209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:24.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:24.222726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:24.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:24.236284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:34.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:34.451060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:34.466 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:34.463575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:44.685 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:44.682607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:44.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:44.691611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:54.954 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:54.951751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:26:54.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:26:54.963502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:05.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:05.223484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:05.238 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:05.236401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:15.459 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:15.457114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:15.471 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:15.468854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:25.692 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:25.690113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:25.706 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:25.703418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:35.925 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:35.922626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:35.939 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:35.936931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:46.160 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:46.157766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:46.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:46.166945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:56.386 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:56.382684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:27:56.398 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:27:56.396360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:06.615 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:06.611815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:06.627 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:06.624963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:16.847 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:16.845108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:16.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:16.855026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:27.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:27.074400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:27.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:27.084884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:37.306 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:37.303608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:37.318 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:37.315681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:47.533 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:47.530459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:47.545 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:47.543265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:57.766 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:57.763625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:28:57.777 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:28:57.774702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:07.997 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:07.994940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:08.007 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:08.004837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:18.230 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:18.227834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:18.244 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:18.240303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:28.462 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:28.459303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:28.474 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:28.471512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:38.700 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:38.697207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:38.713 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:38.711150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:48.944 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:48.933936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:48.952 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:48.950027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:59.179 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:59.176557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:29:59.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:29:59.189539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:09.425 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:09.422561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:09.433 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:09.430884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:19.646 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:19.643436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:19.657 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:19.654795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:29.906 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:29.903417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:29.916 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:29.913331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:40.137 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:40.134653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:40.150 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:40.147429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:50.395 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:50.392671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:30:50.410 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:30:50.404978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:00.649 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:00.646580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:00.658 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:00.654940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:10.975 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:10.972655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:10.987 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:10.984992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:21.254 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:21.251137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:21.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:21.263995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:31.485 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:31.481636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:31.496 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:31.493890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:41.717 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:41.714548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:41.729 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:41.726502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:51.952 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:51.949707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:31:51.965 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:31:51.962794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:02.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:02.179536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:02.194 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:02.191697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:12.416 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:12.413406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:12.432 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:12.429281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:22.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:22.651122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:22.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:22.662993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:32.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:32.885626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:32.898 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:32.895222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:43.122 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:43.119666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:43.135 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:43.132601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:53.358 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:53.355456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:32:53.367 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:32:53.364897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:03.586 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:03.583398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:03.598 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:03.595862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:09.566 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:09.563327 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T03:33:10.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:10.188860 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:33:10.192 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:33:10.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:33:10.193 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:33:10.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:10.188881 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:33:10.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:33:10.194 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:33:10.181+0000 7f3cb0cb8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:33:10.194 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:33:13.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:13.850307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:13.866 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:13.863648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:24.090 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:24.087305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:24.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:24.100811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:34.326 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:34.323025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:34.335 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:34.332811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:44.556 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:44.553316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:44.572 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:44.566825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:54.792 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:54.789923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:33:54.801 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:33:54.798905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:05.016 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:05.013176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:05.076 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:05.072655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:05.411 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:05.408460 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 03:34:05.398467286 +0000 UTC m=+10853.414438018 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T03:34:15.295 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:15.292086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:15.309 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:15.306795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:25.551 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:25.548850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:25.562 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:25.560039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:35.818 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:35.814844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:35.830 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:35.827150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:46.069 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:46.065803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:46.080 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:46.078135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:56.305 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:56.301053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:34:56.315 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:34:56.312768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:06.585 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:06.581195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:06.595 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:06.591986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:16.811 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:16.808361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:16.823 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:16.820792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:27.044 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:27.041292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:27.062 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:27.055403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:37.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:37.279131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:37.294 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:37.291137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:47.513 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:47.509879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:47.524 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:47.522142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:57.762 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:57.759136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:35:57.775 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:35:57.771978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:07.992 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:07.988839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:08.004 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:08.001436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:18.224 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:18.220862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:18.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:18.233902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:28.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:28.452666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:28.468 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:28.465053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:38.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:38.692198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:38.708 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:38.705460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:48.927 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:48.923837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:48.939 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:48.936047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:59.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:59.162207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:36:59.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:36:59.179023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:09.404 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:09.401089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:09.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:09.461003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:19.681 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:19.677714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:19.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:19.690819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:29.915 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:29.912081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:29.928 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:29.925724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:40.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:40.145130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:40.160 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:40.157101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:50.380 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:50.376957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:37:50.392 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:37:50.388896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:00.609 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:00.606237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:00.620 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:00.617708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:10.845 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:10.841933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:10.857 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:10.853915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:21.076 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:21.071918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:21.088 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:21.084249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:31.335 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:31.330286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:31.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:31.347580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:41.569 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:41.565559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:41.582 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:41.579260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:51.807 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:51.804300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:38:51.818 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:38:51.815200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:02.050 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:02.046655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:02.062 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:02.058808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:12.316 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:12.312801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:12.327 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:12.324776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:22.579 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:22.575415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:22.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:22.589865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:32.811 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:32.808158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:32.826 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:32.822794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:43.074 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:43.071018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:43.088 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:43.085122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:53.308 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:53.305161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:39:53.317 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:39:53.314230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:03.534 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:03.530993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:03.547 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:03.544646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:13.771 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:13.767916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:13.780 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:13.777056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:23.999 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:23.996252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:24.011 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:24.007546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:34.231 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:34.227833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:34.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:34.244286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:44.465 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:44.462478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:44.549 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:44.545169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:54.776 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:54.773265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:40:54.791 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:40:54.788278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:05.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:05.008807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:05.024 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:05.021626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:15.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:15.245824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:15.259 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:15.256597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:25.481 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:25.477493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:25.493 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:25.490584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:35.714 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:35.709735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:35.725 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:35.722013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:45.946 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:45.941931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:45.959 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:45.956072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:56.177 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:56.173636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:41:56.198 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:41:56.195652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:06.415 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:06.411282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:06.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:06.435179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:16.725 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:16.722255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:16.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:16.734962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:26.957 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:26.954050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:26.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:26.966397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:37.190 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:37.186951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:37.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:37.200335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:47.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:47.420379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:47.435 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:47.432258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:57.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:57.651224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:42:57.668 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:42:57.664809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:07.884 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:07.880626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:07.917 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:07.913972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:18.159 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:18.155765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:18.168 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:18.164573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:28.383 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:28.379387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:28.400 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:28.396833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:38.620 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:38.616957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:38.630 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:38.627434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:48.851 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:48.845715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:48.871 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:48.862126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:59.084 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:59.080892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:43:59.099 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:43:59.096203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:09.323 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:09.318742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:09.335 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:09.330242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:19.556 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:19.552891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:19.568 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:19.564997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:29.788 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:29.785057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:29.800 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:29.797571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:40.022 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:40.018900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:40.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:40.032599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:50.258 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:50.255300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:44:50.270 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:44:50.267147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:00.533 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:00.529433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:00.544 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:00.540739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:10.767 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:10.762044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:10.778 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:10.774865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:21.000 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:20.996615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:21.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:21.007601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:31.230 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:31.227362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:31.241 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:31.238346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:41.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:41.460226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:41.479 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:41.474427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:51.697 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:51.694719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:45:51.711 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:45:51.708605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:01.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:01.933100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:01.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:01.946255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:12.189 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:12.186611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:12.197 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:12.194946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:22.418 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:22.413918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:22.467 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:22.461924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:32.720 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:32.717930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:32.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:32.729974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:42.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:42.947686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:42.963 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:42.960896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:53.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:53.179885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:46:53.193 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:46:53.190755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:03.431 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:03.428727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:03.444 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:03.442161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:13.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:13.662615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:13.689 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:13.686469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:23.936 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:23.933492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:23.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:23.947945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:34.174 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:34.171328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:34.185 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:34.183117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:44.406 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:44.403342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:44.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:44.414940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:54.639 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:54.637166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:47:54.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:47:54.649400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:04.874 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:04.871462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:04.886 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:04.883309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:15.109 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:15.106472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:15.122 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:15.120040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:25.351 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:25.349015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:25.368 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:25.366056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:35.643 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:35.640187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:35.654 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:35.652214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:45.870 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:45.867614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:45.882 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:45.880389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:56.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:56.101211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:48:56.116 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:48:56.113534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:06.340 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:06.337533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:06.351 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:06.348826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:16.567 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:16.565577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:16.578 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:16.576570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:26.804 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:26.802168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:26.812 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:26.810392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:37.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:37.033104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:37.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:37.046342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:47.265 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:47.263005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:47.278 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:47.276472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:50.199 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:50.197725 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T03:49:50.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:50.826582 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:49:50.819+0000 7f345272b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:49:50.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:49:50.819+0000 7f345272b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:49:50.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:49:50.819+0000 7f345272b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:49:50.830 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:49:50.830 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:50.826614 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:49:50.819+0000 7f345272b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:49:50.830 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:49:50.819+0000 7f345272b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:49:50.831 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T03:49:50.819+0000 7f345272b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:49:50.831 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:49:57.498 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:57.496252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:49:57.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:49:57.507463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:07.736 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:07.734800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:07.749 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:07.747325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:17.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:17.970655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:17.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:17.982712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:28.206 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:28.204488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:28.218 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:28.216823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:38.439 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:38.437047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:38.451 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:38.449515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:48.667 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:48.665998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:48.678 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:48.676777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:58.926 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:58.924888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:50:58.988 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:50:58.986189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:09.224 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:09.222888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:09.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:09.234974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:19.503 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:19.499341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:19.512 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:19.510622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:29.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:29.729527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:29.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:29.742918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:39.972 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:39.970446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:39.983 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:39.981824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:50.210 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:50.208169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:51:50.219 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:51:50.217347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:00.440 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:00.437454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:00.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:00.450571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:10.672 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:10.669716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:10.683 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:10.682521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:20.905 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:20.903620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:20.917 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:20.915643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:21.076 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:21.074520 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 03:52:21.064327819 +0000 UTC m=+11949.080298579 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T03:52:31.135 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:31.133503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:31.197 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:31.196073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:41.423 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:41.422185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:41.437 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:41.435763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:51.699 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:51.698116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:52:51.713 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:52:51.711942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:01.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:01.948596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:01.962 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:01.960657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:12.195 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:12.193910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:12.208 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:12.206815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:22.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:22.423276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:22.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:22.437406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:32.659 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:32.658288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:32.668 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:32.667018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:42.888 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:42.887320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:42.902 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:42.900685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:53.121 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:53.120160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:53:53.133 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:53:53.132504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:03.355 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:03.354271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:03.367 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:03.364879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:13.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:13.591186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:13.605 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:13.602527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:23.829 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:23.828398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:23.842 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:23.841121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:34.066 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:34.065381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:34.077 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:34.076750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:44.297 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:44.295779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:44.306 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:44.305289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:54.529 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:54.527875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:54:54.536 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:54:54.535462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:04.760 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:04.758929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:04.774 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:04.773107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:14.993 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:14.991762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:15.004 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:15.003246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:25.221 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:25.219869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:25.234 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:25.231281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:35.447 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:35.445595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:35.458 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:35.457487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:45.684 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:45.683296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:45.785 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:45.695022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:55.926 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:55.925524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:55:55.938 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:55:55.937028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:06.196 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:06.195188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:06.203 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:06.202907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:16.421 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:16.420058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:16.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:16.432668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:26.655 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:26.653298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:26.667 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:26.665708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:36.886 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:36.884743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:36.898 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:36.897898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:47.120 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:47.119244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:47.133 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:47.132557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:57.355 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:57.354333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:56:57.364 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:56:57.363245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:07.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:07.580039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:07.594 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:07.593366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:17.813 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:17.812307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:17.825 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:17.824835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:28.045 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:28.044071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:28.058 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:28.056851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:38.303 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:38.303064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:38.315 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:38.314466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:48.550 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:48.549862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:48.562 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:48.561559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:58.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:58.783548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:57:58.795 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:57:58.794402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:09.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:09.016647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:09.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:09.030991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:19.257 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:19.257052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:19.272 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:19.271667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:29.490 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:29.488994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:29.551 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:29.549987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:39.814 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:39.813379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:39.825 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:39.825055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:50.179 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:50.177893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:58:50.187 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:58:50.187015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:00.424 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:00.423257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:00.437 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:00.437045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:10.664 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:10.663212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:10.675 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:10.674419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:20.915 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:20.915054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:20.927 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:20.926952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:31.153 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:31.152059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:31.167 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:31.167079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:41.382 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:41.381757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:41.395 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:41.395131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:51.628 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:51.627687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T03:59:51.636 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 03:59:51.635785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:01.853 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:01.852619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:01.865 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:01.864957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:12.089 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:12.087932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:12.101 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:12.100703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:22.347 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:22.346312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:22.359 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:22.358853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:32.583 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:32.582512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:32.595 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:32.594882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:42.819 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:42.818951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:42.831 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:42.830662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:53.045 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:53.044392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:00:53.057 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:00:53.055897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:03.278 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:03.277385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:03.291 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:03.289974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:13.518 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:13.517813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:13.530 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:13.529182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:23.750 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:23.749598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:23.763 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:23.762860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:33.999 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:33.998936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:34.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:34.012134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:44.265 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:44.264399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:44.279 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:44.278970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:54.545 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:54.543769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:01:54.555 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:01:54.555356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:04.781 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:04.779684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:04.792 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:04.791749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:15.010 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:15.009738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:15.019 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:15.018737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:25.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:25.249100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:25.262 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:25.262261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:35.479 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:35.478523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:35.492 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:35.490852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:45.734 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:45.733682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:45.745 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:45.745164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:56.123 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:56.122881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:02:56.148 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:02:56.148408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:06.384 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:06.384285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:06.395 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:06.395277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:16.615 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:16.614709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:16.695 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:16.694297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:26.957 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:26.957025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:26.969 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:26.968782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:37.187 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:37.186944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:37.202 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:37.202027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:47.421 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:47.420737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:47.434 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:47.433911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:57.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:57.651158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:03:57.665 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:03:57.664836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:07.883 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:07.882207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:07.894 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:07.893847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:18.117 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:18.116754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:18.130 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:18.130199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:28.353 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:28.353061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:28.364 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:28.364502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:38.587 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:38.586998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:38.600 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:38.600204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:48.827 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:48.826876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:48.837 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:48.837777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:59.063 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:59.063227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:04:59.075 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:04:59.075428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:09.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:09.299063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:09.311 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:09.311611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:19.538 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:19.537388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:19.551 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:19.550907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:29.780 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:29.779603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:29.794 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:29.793672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:40.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:40.030155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:40.042 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:40.041669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:50.267 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:50.266792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:05:50.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:05:50.280634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:00.515 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:00.515023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:00.527 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:00.526975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:10.751 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:10.750663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:10.759 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:10.759374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:20.977 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:20.976501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:20.989 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:20.988949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:30.833 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:30.833179 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T04:06:31.201 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:31.201259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:31.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:31.213572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:31.454 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:31.453798 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:06:31.445+0000 7f837f16c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:06:31.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:06:31.445+0000 7f837f16c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:06:31.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:06:31.445+0000 7f837f16c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:06:31.455 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:06:31.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:31.453827 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:06:31.445+0000 7f837f16c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:06:31.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:06:31.445+0000 7f837f16c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:06:31.456 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:06:31.445+0000 7f837f16c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:06:31.456 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:06:41.432 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:41.431428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:41.444 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:41.444229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:51.663 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:51.663023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:06:51.675 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:06:51.674199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:01.893 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:01.893258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:01.913 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:01.912682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:12.172 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:12.171455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:12.518 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:12.185317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:22.407 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:22.407308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:22.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:22.417134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:32.632 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:32.632047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:32.645 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:32.645108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:42.867 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:42.864975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:42.877 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:42.876687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:42.877 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:42.876740 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:53.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:53.095336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:07:53.107 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:07:53.107558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:03.346 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:03.345892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:03.357 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:03.357948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:13.590 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:13.589720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:13.602 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:13.602275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:23.822 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:23.821829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:23.834 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:23.833782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:34.054 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:34.053788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:34.065 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:34.065110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:44.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:44.287980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:44.299 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:44.298128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:54.538 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:54.537556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:08:54.553 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:08:54.553172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:04.803 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:04.802505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:04.814 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:04.813950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:15.036 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:15.035721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:15.047 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:15.046944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:25.274 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:25.274187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:25.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:25.286571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:35.509 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:35.509345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:35.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:35.521309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:35.867 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:35.866578 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 04:09:35.860227605 +0000 UTC m=+12983.876198308 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:09:45.747 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:45.747098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:45.760 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:45.760607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:55.979 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:55.979191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:09:55.993 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:09:55.992696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:06.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:06.215970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:06.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:06.225457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:16.451 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:16.450572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:16.495 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:16.494962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:26.751 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:26.747465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:26.760 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:26.759588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:37.031 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:37.031490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:37.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:37.047378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:47.268 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:47.268195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:47.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:47.279915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:57.501 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:57.500835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:10:57.570 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:10:57.570111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:07.786 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:07.786202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:07.798 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:07.798543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:18.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:18.035155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:18.051 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:18.051905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:28.270 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:28.270065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:28.282 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:28.282584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:38.540 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:38.539943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:38.553 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:38.553173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:48.771 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:48.770993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:48.784 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:48.784107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:59.002 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:59.001859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:11:59.012 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:11:59.013153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:09.275 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:09.274798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:09.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:09.286458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:19.503 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:19.502749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:19.517 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:19.517547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:29.734 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:29.733188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:29.746 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:29.745618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:39.966 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:39.965629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:39.981 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:39.980651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:50.202 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:50.200325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:12:50.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:12:50.211379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:00.429 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:00.429452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:00.442 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:00.442807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:10.666 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:10.666588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:10.680 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:10.679690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:20.899 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:20.899207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:20.912 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:20.912127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:31.129 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:31.129197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:31.141 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:31.141685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:41.362 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:41.362053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:41.376 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:41.373905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:51.600 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:51.599418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:13:51.608 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:13:51.608613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:01.828 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:01.828120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:01.840 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:01.840225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:12.061 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:12.060703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:12.076 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:12.077048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:22.350 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:22.350026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:22.361 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:22.361143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:32.581 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:32.580929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:32.593 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:32.593860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:42.815 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:42.815372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:42.827 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:42.827493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:53.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:53.047770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:14:53.059 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:14:53.059659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:03.277 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:03.276545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:03.288 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:03.287527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:13.551 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:13.551815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:13.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:13.560643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:23.782 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:23.782438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:23.793 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:23.793594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:34.007 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:34.007057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:34.021 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:34.020916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:44.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:44.247594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:44.260 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:44.260907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:54.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:54.482024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:15:54.494 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:15:54.494058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:04.744 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:04.743718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:04.757 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:04.757572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:14.984 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:14.984526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:14.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:14.996111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:25.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:25.213244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:25.220 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:25.220728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:35.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:35.438330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:35.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:35.454478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:45.670 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:45.670070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:45.691 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:45.691139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:55.915 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:55.914855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:16:55.930 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:16:55.930329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:06.156 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:06.156126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:06.169 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:06.169900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:16.388 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:16.387672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:16.399 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:16.399671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:26.620 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:26.619826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:26.633 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:26.633090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:36.858 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:36.858261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:36.868 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:36.868677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:47.092 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:47.092763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:47.103 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:47.103378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:57.324 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:57.324101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:17:57.332 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:17:57.332716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:07.555 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:07.554055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:07.575 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:07.575310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:17.807 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:17.806549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:17.818 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:17.818624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:28.038 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:28.038402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:28.052 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:28.052623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:38.271 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:38.271588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:38.283 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:38.283458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:48.502 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:48.501893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:48.516 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:48.515833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:58.737 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:58.737296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:18:58.748 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:18:58.746278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:08.966 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:08.966031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:08.978 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:08.978192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:19.197 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:19.196801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:19.247 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:19.246968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:29.467 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:29.466130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:29.479 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:29.478461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:39.700 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:39.699728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:39.713 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:39.713299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:49.946 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:49.945718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:19:49.957 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:19:49.957518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:00.175 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:00.175019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:00.182 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:00.182480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:10.404 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:10.404569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:10.417 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:10.417110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:20.632 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:20.632505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:20.643 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:20.643336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:30.860 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:30.860281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:30.874 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:30.874450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:41.097 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:41.096763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:41.110 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:41.111073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:51.332 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:51.332410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:20:51.340 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:20:51.341029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:01.563 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:01.563045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:01.571 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:01.571712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:11.792 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:11.791939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:11.805 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:11.805047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:22.026 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:22.026321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:22.038 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:22.038912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:32.264 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:32.264333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:32.276 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:32.276518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:42.502 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:42.502372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:42.514 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:42.514865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:52.739 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:52.739417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:21:52.756 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:21:52.756642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:02.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:02.995809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:03.007 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:03.007589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:13.225 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:13.224923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:13.237 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:13.237726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:23.455 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:23.454805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:23.515 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:23.512898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:33.821 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:33.821613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:33.834 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:33.835043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:44.055 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:44.054544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:44.066 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:44.066171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:54.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:54.285314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:22:54.300 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:22:54.299573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:04.521 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:04.521816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:04.534 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:04.534689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:11.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:11.463257 I | op-mon: parsing mon endpoints: a=10.98.3.151:6789 2022-03-08T04:23:12.105 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:12.104540 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:23:12.095+0000 7fe3043df340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:23:12.105 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:23:12.095+0000 7fe3043df340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:23:12.105 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:23:12.095+0000 7fe3043df340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:23:12.106 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:23:12.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:12.104568 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:23:12.095+0000 7fe3043df340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:23:12.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:23:12.095+0000 7fe3043df340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:23:12.106 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08T04:23:12.095+0000 7fe3043df340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:23:12.107 INFO:tasks.rook.operator.smithi120.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:23:14.761 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:14.761393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:14.769 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:14.770358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:24.991 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:24.991509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:25.001 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:25.001733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:35.217 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:35.217817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:35.228 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:35.228480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:45.481 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:45.481321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:45.491 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:45.491775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:55.712 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:55.712843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:23:55.724 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:23:55.724483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:05.986 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:05.986399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:06.033 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:06.033783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:16.249 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:16.249119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:16.259 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:16.260347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:26.549 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:26.549203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:26.560 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:26.560958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:36.968 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:36.968107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:36.976 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:36.976800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:47.280 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:47.280280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:47.289 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:47.290053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:57.585 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:57.584106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:24:57.601 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:24:57.597735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:07.983 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:07.983997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:07.996 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:07.995445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:18.213 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:18.213701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:18.226 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:18.226475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:28.463 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:28.463801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:28.473 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:28.473506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:38.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:38.693530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:38.838 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:38.838540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:49.083 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:49.083271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:49.093 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:49.093738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:59.358 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:59.358940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:25:59.413 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:25:59.413639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:09.631 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:09.630889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:09.648 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:09.644071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:19.867 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:19.865190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:19.878 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:19.878230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:30.099 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:30.099895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:30.115 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:30.115376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:40.333 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:40.333881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:40.341 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:40.342160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:50.557 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:50.558158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:50.568 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:50.568761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:26:50.789 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:26:50.789481 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 04:26:50.777050399 +0000 UTC m=+14018.793021138 LastTransitionTime:2022-03-08 00:35:13 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:27:00.792 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:00.792040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:00.805 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:00.805562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:11.060 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:11.060660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:11.073 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:11.073749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:21.334 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:21.334106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:21.347 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:21.347311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:31.569 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:31.569249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:31.582 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:31.582967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:41.817 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:41.817404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:41.828 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:41.828755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:52.131 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:52.131415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:27:52.145 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:27:52.145508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:02.399 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:02.399789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:02.438 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:02.407747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:12.629 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:12.629445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:12.643 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:12.641556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:22.861 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:22.861500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:22.876 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:22.876583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:33.095 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:33.095993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:33.108 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:33.108884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:43.328 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:43.328894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:43.338 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:43.339063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:53.559 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:53.559962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:28:53.574 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:28:53.573613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:03.796 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:03.796902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:03.810 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:03.810508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:14.035 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:14.035805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:14.048 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:14.048965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:24.273 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:24.273475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:24.286 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:24.287132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:34.505 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:34.504515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:34.518 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:34.518126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:44.738 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:44.738200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:44.783 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:44.783492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:55.004 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:55.004865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:29:55.017 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:29:55.018000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:05.236 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:05.235930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:05.248 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:05.248122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:15.469 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:15.469433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:15.483 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:15.483230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:25.702 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:25.702568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:25.712 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:25.713098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:35.937 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:35.937858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:35.950 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:35.951175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:46.211 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:46.211461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:46.259 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:46.257262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:56.473 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:56.474095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:30:56.485 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:30:56.485854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:06.709 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:06.709433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:06.722 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:06.722612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:16.941 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:16.941071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:16.954 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:16.954317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:27.173 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:27.173907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:27.186 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:27.185917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:37.421 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:37.421510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:37.433 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:37.433811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:47.651 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:47.651190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:47.688 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:47.686691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:57.908 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:57.909058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:31:57.919 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:31:57.920134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:08.143 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:08.143364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:08.163 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:08.164217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:18.431 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:18.432087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:18.477 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:18.477513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:28.712 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:28.713347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:28.732 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:28.732774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:38.957 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:38.957224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:38.970 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:38.970518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:49.216 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:49.216100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:49.229 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:49.229369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:59.453 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:59.453160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:32:59.464 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:32:59.465352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:33:09.684 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:33:09.682689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T04:33:09.693 INFO:tasks.rook.operator.smithi120.stdout:2022-03-08 04:33:09.694382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T06:53:59.304 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-08T06:53:59.333 DEBUG:teuthology.task.console_log:Killing console logger for smithi120 2022-03-08T06:53:59.334 DEBUG:teuthology.exit:Finished running handlers