2022-02-21T18:20:23.171 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-21T18:20:23.172 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542 -- /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542/orig.config.yaml 2022-02-21T18:20:23.212 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-02-21T18:20:23.284 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542 branch: wip-yuri4-testing-2022-02-18-0800 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6698542' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2022-02-18-0800 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri4-testing-2022-02-18-0800 sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri4-testing-2022-02-18-0800 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e targets: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDG5Ft1erwvE/+RD4GpWXLdJgbHO4vyVKj7UGpAtUSBctkrepndBmvthv4cf/HPau9g0oiFs9AI+xSIPOa90tU2u0MUgleWGlYfesNOhewm0jc/Z0D7VbvJMDjsgPbuqWNE+Vs6wMUgzhL4Y8tWHJkO8zD37z/dOfWFWvY4xDdEGZvfGSvP/B1uc26IT5spx4xMXCQpQvkvhFntPXDNTSqctCM875dh1ASBvqqBQcdYvOtJF4zzsfhDKmUF8uHA9pgDRRtF1ZnvZo+iKbMwpOtUqmF2ks7V3lv50y0ZfxnK0nfC0Ghft7ErHXdC8xXHXkkGrtrhRcL3Tkp7Vj8ReLDcXMMBJoPIp1HwA20LK/3urcXmMdtvcg8pW4bpfA97kkgVvngO7CXGzHBVstKsauSnXfVuHb1tQtjoNE+vr6fp5zu6JCCtImbQ5HW3WfAXu1TKaYG9HeNOaI8Lx1fWO/SNUdB+aBUJ21PsrF1vwGZ3a22A+yozzJDyty4toGzysak= smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzQwTOCot+aSLwIgRU7AhJEKYzdId64wBLXk0Vht51DOrk5cNjT7X7tHGX9BIrS8et6TXnUytDBXhBB5tTazV5jjUHjNyEOH8/H7tSUxhplLZCix0LGiS+6ln1FrVWQVlM73SF5v3NqhvA9s8j2t9yWaGWW5ykn0qG+bFfQ7TtLh5TybGwaLjedvdyLDyM+hv+VPnFOEsCXk/6zVh3YsUmCCy9xI8hE5Rgborrg7WCXz9hmevzCHiUuh3O612z4owjOTbM2va/Z7Q+WHq/uh+IRrBhd9cslmBAuBAyF+IacYqCxn4Ulx++CORbONTBsMirIcnwzCoKPQGEmtLMmXCEOLiDMfPspDxQT0Is0XP5fm9/omJ9JSBERFMv/VGjVNtCLtbP6mr7M5oczU5fUQypnTPqw2MOhYFB5QKo2Qq3+kalDNhSsPbcFWVrl4/fvJLUo5VXlaMaKEPcCDPQ3AwWhdjj11ZuMr+OvPZT1ig9e1k+bfSX4Yl5llG+qMqButM= smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDxOls2VFPWvFFnciooMT8DvX6q2pXrwTjEklLtIOtpsH83teumdKQIOgR4JhJwBkXr8/XfzZNhrhUbgVphmhcKE2sts52wbW+Ns8p4lrVfJNgzDJ3xSfdiPgxxj7l3DscJedbmZ5PxPkoENcNVm0o0LwreapSc/h6G4rPE7CnKTdAzhaq6VmdZEYhwD1x9mqJKFevXKVjPAGF0zYoHYjs+l2WPH+BHRaLIxoWLmNhSQ0Ry3bhMrOrheVZfVYq2ERM8OZDVsX485ryTDnoCvICeJ6G0Ng6Wdb8r0OBwSKfL0Kobi+v4adacOoZtCgYC67/66nhdtZM11C6Tum9fPAzWImuvsLK3uTaGyK8yphs/l7ZjhOsqmtum/pkdaJqiBrPN+5gvElhcfI+7EGSkG6uTj9+wBOCohfMPFNhVeNiT3IO56dlrZStI2eGIiigYrMzfGHuYhkLJCjKO1ulPe/e+8NgnUGv+T70YnInvEgFrkhgzyEmTdFVXvxWNTmnJdE= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eea2521245e542c7c1a063d296779f572aa0255a timestamp: 2022-02-21_15:40:41 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 2022-02-21T18:20:23.284 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa; will attempt to use it 2022-02-21T18:20:23.285 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks 2022-02-21T18:20:23.287 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-02-21T18:20:23.290 INFO:teuthology.task.internal:Checking packages... 2022-02-21T18:20:23.315 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b05010b68efc838fdc35b41106e2ebdabe395a3e' 2022-02-21T18:20:23.315 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-02-21T18:20:23.316 INFO:teuthology.packaging:ref: None 2022-02-21T18:20:23.316 INFO:teuthology.packaging:tag: None 2022-02-21T18:20:23.316 INFO:teuthology.packaging:branch: wip-yuri4-testing-2022-02-18-0800 2022-02-21T18:20:23.317 INFO:teuthology.packaging:sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e 2022-02-21T18:20:23.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2022-02-18-0800 2022-02-21T18:20:23.545 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10781-gb05010b6-1focal 2022-02-21T18:20:23.545 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-02-21T18:20:23.557 INFO:teuthology.task.internal:no buildpackages task found 2022-02-21T18:20:23.557 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-02-21T18:20:23.605 INFO:teuthology.task.internal:Saving configuration 2022-02-21T18:20:23.619 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-02-21T18:20:23.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-21T18:20:23.662 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 18:14:48.976202', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN74lUnH8yzRVEF7Bwg8s5gTsvedcB1ox+yB7V7AFrna'} 2022-02-21T18:20:23.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 18:14:48.975121', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILkpR/mUT2Pt/kPc90Pjsk1BfDEf0oPaILjDMZXp4i0M'} 2022-02-21T18:20:23.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698542', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 18:14:48.973243', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEJ77EDooDSv4eBIWynzlIA59R27io8o7+IMqgcAm6Gv'} 2022-02-21T18:20:23.708 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-02-21T18:20:23.718 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-02-21T18:20:23.718 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-02-21T18:20:23.719 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-02-21T18:20:23.719 INFO:teuthology.run_tasks:Running task console_log... 2022-02-21T18:20:23.883 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1100cbdea0>, signals=[15]) 2022-02-21T18:20:23.884 INFO:teuthology.run_tasks:Running task internal.connect... 2022-02-21T18:20:23.895 INFO:teuthology.task.internal:Opening connections... 2022-02-21T18:20:23.897 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2022-02-21T18:20:23.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:20:23.952 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi140.front.sepia.ceph.com 2022-02-21T18:20:23.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:20:24.004 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2022-02-21T18:20:24.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:20:24.059 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-02-21T18:20:24.070 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2022-02-21T18:20:24.078 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2022-02-21T18:20:24.079 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2022-02-21T18:20:24.128 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2022-02-21T18:20:24.129 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T18:20:24.129 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2022-02-21T18:20:24.129 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2022-02-21T18:20:24.130 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T18:20:24.130 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="20.04" 2022-02-21T18:20:24.130 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T18:20:24.130 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T18:20:24.130 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T18:20:24.131 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T18:20:24.131 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=focal 2022-02-21T18:20:24.131 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=focal 2022-02-21T18:20:24.132 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-02-21T18:20:24.157 DEBUG:teuthology.orchestra.run.smithi140:> uname -m 2022-02-21T18:20:24.165 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64 2022-02-21T18:20:24.166 DEBUG:teuthology.orchestra.run.smithi140:> cat /etc/os-release 2022-02-21T18:20:24.214 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="Ubuntu" 2022-02-21T18:20:24.214 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T18:20:24.214 INFO:teuthology.orchestra.run.smithi140.stdout:ID=ubuntu 2022-02-21T18:20:24.214 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE=debian 2022-02-21T18:20:24.215 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T18:20:24.215 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="20.04" 2022-02-21T18:20:24.215 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T18:20:24.215 INFO:teuthology.orchestra.run.smithi140.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T18:20:24.216 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T18:20:24.216 INFO:teuthology.orchestra.run.smithi140.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T18:20:24.216 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_CODENAME=focal 2022-02-21T18:20:24.216 INFO:teuthology.orchestra.run.smithi140.stdout:UBUNTU_CODENAME=focal 2022-02-21T18:20:24.217 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2022-02-21T18:20:24.236 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2022-02-21T18:20:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2022-02-21T18:20:24.243 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2022-02-21T18:20:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="Ubuntu" 2022-02-21T18:20:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T18:20:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:ID=ubuntu 2022-02-21T18:20:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE=debian 2022-02-21T18:20:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T18:20:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="20.04" 2022-02-21T18:20:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T18:20:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T18:20:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T18:20:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T18:20:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_CODENAME=focal 2022-02-21T18:20:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:UBUNTU_CODENAME=focal 2022-02-21T18:20:24.253 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2022-02-21T18:20:24.271 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-02-21T18:20:24.286 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-02-21T18:20:24.307 INFO:teuthology.task.internal:Checking for old test directory... 2022-02-21T18:20:24.308 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2022-02-21T18:20:24.311 DEBUG:teuthology.orchestra.run.smithi140:> test '!' -e /home/ubuntu/cephtest 2022-02-21T18:20:24.314 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2022-02-21T18:20:24.320 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-02-21T18:20:24.331 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-02-21T18:20:24.332 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2022-02-21T18:20:24.361 DEBUG:teuthology.orchestra.run.smithi140:> test -z $(ls -A /var/lib/ceph) 2022-02-21T18:20:24.365 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2022-02-21T18:20:24.386 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-02-21T18:20:24.466 INFO:teuthology.run_tasks:Running task kernel... 2022-02-21T18:20:24.486 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-02-21T18:20:24.487 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-02-21T18:20:24.487 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-02-21T18:20:24.487 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-02-21T18:20:24.488 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2022-02-21T18:20:24.489 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2022-02-21T18:20:24.489 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2022-02-21T18:20:24.496 INFO:teuthology.orchestra.run.smithi140.stdout:5.4.0-97-generic 2022-02-21T18:20:24.497 INFO:teuthology.task.kernel:Running kernel on smithi140: 5.4.0-97-generic 2022-02-21T18:20:24.497 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get clean 2022-02-21T18:20:24.499 INFO:teuthology.orchestra.run.smithi179.stdout:5.4.0-97-generic 2022-02-21T18:20:24.500 INFO:teuthology.orchestra.run.smithi081.stdout:5.4.0-97-generic 2022-02-21T18:20:24.500 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.4.0-97-generic 2022-02-21T18:20:24.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get clean 2022-02-21T18:20:24.501 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.4.0-97-generic 2022-02-21T18:20:24.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2022-02-21T18:20:24.568 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get update 2022-02-21T18:20:24.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2022-02-21T18:20:24.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2022-02-21T18:20:24.748 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T18:20:24.759 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T18:20:24.807 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T18:20:24.850 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:20:24.867 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:20:24.915 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:20:24.933 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T18:20:24.950 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T18:20:24.998 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T18:20:25.036 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T18:20:25.071 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T18:20:25.133 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T18:20:25.173 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T18:20:25.179 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T18:20:25.186 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T18:20:25.191 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T18:20:25.192 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T18:20:25.192 INFO:teuthology.orchestra.run.smithi140.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T18:20:25.222 INFO:teuthology.orchestra.run.smithi140.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T18:20:25.224 INFO:teuthology.orchestra.run.smithi140.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T18:20:25.225 INFO:teuthology.orchestra.run.smithi140.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T18:20:25.238 INFO:teuthology.orchestra.run.smithi140.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T18:20:25.254 INFO:teuthology.orchestra.run.smithi140.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T18:20:25.256 INFO:teuthology.orchestra.run.smithi140.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T18:20:25.272 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T18:20:25.277 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T18:20:25.278 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T18:20:25.281 INFO:teuthology.orchestra.run.smithi140.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T18:20:25.282 INFO:teuthology.orchestra.run.smithi140.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T18:20:25.283 INFO:teuthology.orchestra.run.smithi140.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T18:20:25.284 INFO:teuthology.orchestra.run.smithi140.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:25.315 INFO:teuthology.orchestra.run.smithi140.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T18:20:25.352 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T18:20:25.356 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T18:20:25.357 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T18:20:25.359 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T18:20:25.359 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T18:20:25.366 INFO:teuthology.orchestra.run.smithi179.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T18:20:25.369 INFO:teuthology.orchestra.run.smithi179.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T18:20:25.371 INFO:teuthology.orchestra.run.smithi179.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T18:20:25.372 INFO:teuthology.orchestra.run.smithi179.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T18:20:25.373 INFO:teuthology.orchestra.run.smithi179.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T18:20:25.402 INFO:teuthology.orchestra.run.smithi179.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T18:20:25.403 INFO:teuthology.orchestra.run.smithi179.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:25.411 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T18:20:25.412 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T18:20:25.423 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T18:20:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T18:20:25.439 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T18:20:25.448 INFO:teuthology.orchestra.run.smithi140.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T18:20:25.471 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T18:20:25.472 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T18:20:25.484 INFO:teuthology.orchestra.run.smithi179.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T18:20:25.485 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T18:20:25.486 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T18:20:25.502 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T18:20:25.535 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T18:20:25.538 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T18:20:25.538 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T18:20:25.539 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T18:20:25.561 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T18:20:25.561 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T18:20:25.561 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:25.817 INFO:teuthology.orchestra.run.smithi179.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T18:20:25.851 INFO:teuthology.orchestra.run.smithi140.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T18:20:26.035 INFO:teuthology.orchestra.run.smithi140.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T18:20:26.045 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T18:20:26.046 INFO:teuthology.orchestra.run.smithi140.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T18:20:26.046 INFO:teuthology.orchestra.run.smithi140.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T18:20:26.047 INFO:teuthology.orchestra.run.smithi140.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T18:20:26.102 INFO:teuthology.orchestra.run.smithi140.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T18:20:26.114 INFO:teuthology.orchestra.run.smithi140.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:26.114 INFO:teuthology.orchestra.run.smithi140.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T18:20:26.120 INFO:teuthology.orchestra.run.smithi179.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T18:20:26.127 INFO:teuthology.orchestra.run.smithi179.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T18:20:26.127 INFO:teuthology.orchestra.run.smithi179.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T18:20:26.128 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T18:20:26.128 INFO:teuthology.orchestra.run.smithi179.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T18:20:26.130 INFO:teuthology.orchestra.run.smithi140.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T18:20:26.133 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T18:20:26.134 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T18:20:26.135 INFO:teuthology.orchestra.run.smithi140.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T18:20:26.137 INFO:teuthology.orchestra.run.smithi140.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T18:20:26.185 INFO:teuthology.orchestra.run.smithi179.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T18:20:26.204 INFO:teuthology.orchestra.run.smithi179.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:26.204 INFO:teuthology.orchestra.run.smithi179.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T18:20:26.208 INFO:teuthology.orchestra.run.smithi140.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T18:20:26.208 INFO:teuthology.orchestra.run.smithi140.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T18:20:26.209 INFO:teuthology.orchestra.run.smithi140.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T18:20:26.210 INFO:teuthology.orchestra.run.smithi140.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T18:20:26.210 INFO:teuthology.orchestra.run.smithi140.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T18:20:26.233 INFO:teuthology.orchestra.run.smithi140.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T18:20:26.234 INFO:teuthology.orchestra.run.smithi140.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T18:20:26.241 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T18:20:26.242 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T18:20:26.250 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T18:20:26.251 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T18:20:26.253 INFO:teuthology.orchestra.run.smithi179.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T18:20:26.300 INFO:teuthology.orchestra.run.smithi179.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T18:20:26.349 INFO:teuthology.orchestra.run.smithi140.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T18:20:26.389 INFO:teuthology.orchestra.run.smithi179.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T18:20:26.389 INFO:teuthology.orchestra.run.smithi179.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T18:20:26.390 INFO:teuthology.orchestra.run.smithi179.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T18:20:26.390 INFO:teuthology.orchestra.run.smithi179.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T18:20:26.391 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T18:20:26.392 INFO:teuthology.orchestra.run.smithi179.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T18:20:26.392 INFO:teuthology.orchestra.run.smithi179.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T18:20:26.392 INFO:teuthology.orchestra.run.smithi179.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T18:20:26.400 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T18:20:26.403 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T18:20:26.403 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T18:20:26.403 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T18:20:26.404 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T18:20:26.404 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T18:20:26.413 INFO:teuthology.orchestra.run.smithi179.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T18:20:26.414 INFO:teuthology.orchestra.run.smithi179.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T18:20:26.443 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T18:20:26.444 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T18:20:26.525 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T18:20:26.525 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T18:20:28.432 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 9,819 kB in 2s (5,463 kB/s) 2022-02-21T18:20:28.648 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 9,819 kB in 2s (5,126 kB/s) 2022-02-21T18:20:28.703 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 9,819 kB in 2s (4,991 kB/s) 2022-02-21T18:20:29.597 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:20:29.620 DEBUG:teuthology.orchestra.run.smithi140:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:20:29.685 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:20:29.840 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:20:29.860 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:20:29.860 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:20:29.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:20:29.878 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:20:29.902 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:20:29.930 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:20:29.974 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:20:30.001 INFO:teuthology.orchestra.run.smithi140.stdout:The following additional packages will be installed: 2022-02-21T18:20:30.002 INFO:teuthology.orchestra.run.smithi140.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.002 INFO:teuthology.orchestra.run.smithi140.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T18:20:30.002 INFO:teuthology.orchestra.run.smithi140.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T18:20:30.003 INFO:teuthology.orchestra.run.smithi140.stdout:Suggested packages: 2022-02-21T18:20:30.003 INFO:teuthology.orchestra.run.smithi140.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T18:20:30.003 INFO:teuthology.orchestra.run.smithi140.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T18:20:30.044 INFO:teuthology.orchestra.run.smithi140.stdout:The following NEW packages will be installed: 2022-02-21T18:20:30.045 INFO:teuthology.orchestra.run.smithi140.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.045 INFO:teuthology.orchestra.run.smithi140.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T18:20:30.046 INFO:teuthology.orchestra.run.smithi140.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T18:20:30.046 INFO:teuthology.orchestra.run.smithi140.stdout: thermald upower usbmuxd 2022-02-21T18:20:30.103 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:20:30.313 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:20:30.313 INFO:teuthology.orchestra.run.smithi140.stdout:Need to get 80.1 MB of archives. 2022-02-21T18:20:30.313 INFO:teuthology.orchestra.run.smithi140.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T18:20:30.313 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T18:20:30.315 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:20:30.315 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:20:30.316 INFO:teuthology.orchestra.run.smithi179.stdout:The following additional packages will be installed: 2022-02-21T18:20:30.316 INFO:teuthology.orchestra.run.smithi179.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.316 INFO:teuthology.orchestra.run.smithi179.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T18:20:30.316 INFO:teuthology.orchestra.run.smithi179.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T18:20:30.317 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2022-02-21T18:20:30.317 INFO:teuthology.orchestra.run.smithi179.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T18:20:30.317 INFO:teuthology.orchestra.run.smithi179.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T18:20:30.318 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:20:30.318 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2022-02-21T18:20:30.318 INFO:teuthology.orchestra.run.smithi081.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.319 INFO:teuthology.orchestra.run.smithi081.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T18:20:30.319 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T18:20:30.319 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2022-02-21T18:20:30.320 INFO:teuthology.orchestra.run.smithi081.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T18:20:30.320 INFO:teuthology.orchestra.run.smithi081.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T18:20:30.320 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2022-02-21T18:20:30.320 INFO:teuthology.orchestra.run.smithi081.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.320 INFO:teuthology.orchestra.run.smithi081.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T18:20:30.321 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T18:20:30.321 INFO:teuthology.orchestra.run.smithi081.stdout: thermald upower usbmuxd 2022-02-21T18:20:30.345 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2022-02-21T18:20:30.347 INFO:teuthology.orchestra.run.smithi179.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T18:20:30.347 INFO:teuthology.orchestra.run.smithi179.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T18:20:30.347 INFO:teuthology.orchestra.run.smithi179.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T18:20:30.347 INFO:teuthology.orchestra.run.smithi179.stdout: thermald upower usbmuxd 2022-02-21T18:20:30.515 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T18:20:30.529 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:20:30.530 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 80.1 MB of archives. 2022-02-21T18:20:30.530 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T18:20:30.530 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T18:20:30.573 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T18:20:30.576 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T18:20:30.699 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:20:30.700 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 80.1 MB of archives. 2022-02-21T18:20:30.700 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T18:20:30.700 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T18:20:30.701 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T18:20:30.774 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T18:20:30.785 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T18:20:30.802 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T18:20:30.803 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T18:20:30.887 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T18:20:30.887 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T18:20:30.975 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T18:20:30.995 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T18:20:31.006 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T18:20:31.095 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T18:20:31.122 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T18:20:31.827 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T18:20:31.937 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T18:20:32.115 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T18:20:32.303 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T18:20:32.696 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T18:20:33.048 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T18:20:34.512 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T18:20:34.512 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T18:20:34.513 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T18:20:34.518 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T18:20:34.519 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T18:20:34.757 INFO:teuthology.orchestra.run.smithi140.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T18:20:34.758 INFO:teuthology.orchestra.run.smithi140.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T18:20:34.759 INFO:teuthology.orchestra.run.smithi140.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T18:20:34.764 INFO:teuthology.orchestra.run.smithi140.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T18:20:34.766 INFO:teuthology.orchestra.run.smithi140.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T18:20:34.910 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 80.1 MB in 4s (19.2 MB/s) 2022-02-21T18:20:35.108 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T18:20:35.176 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 80.1 MB in 5s (17.0 MB/s) 2022-02-21T18:20:35.363 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T18:20:35.363 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T18:20:35.364 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T18:20:35.370 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T18:20:35.372 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T18:20:35.384 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T18:20:35.776 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 80.1 MB in 5s (15.8 MB/s) 2022-02-21T18:20:35.967 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T18:20:36.560 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T18:20:36.564 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T18:20:36.689 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:36.878 INFO:teuthology.orchestra.run.smithi140.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T18:20:36.883 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T18:20:37.006 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:37.033 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T18:20:37.048 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T18:20:37.100 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:37.309 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T18:20:37.324 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T18:20:37.367 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:37.428 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T18:20:37.433 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T18:20:37.453 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T18:20:37.466 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T18:20:37.503 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:37.565 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:37.703 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T18:20:37.718 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T18:20:37.761 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:37.831 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T18:20:37.843 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T18:20:37.880 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:37.959 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T18:20:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T18:20:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:38.106 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T18:20:38.120 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T18:20:38.166 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:38.192 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T18:20:38.204 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:38.250 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:38.363 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T18:20:38.376 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T18:20:38.412 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:38.509 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T18:20:38.523 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:38.559 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:38.570 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T18:20:38.584 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:38.628 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:38.707 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T18:20:38.722 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T18:20:38.757 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:38.846 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T18:20:38.861 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:38.903 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:39.118 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T18:20:39.131 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:39.168 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:39.505 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T18:20:39.519 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:39.563 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:40.845 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T18:20:40.861 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:40.959 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:41.187 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T18:20:41.200 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:41.293 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:41.457 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T18:20:41.470 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:41.515 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:41.780 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T18:20:41.796 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:41.798 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T18:20:41.812 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:41.856 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:41.919 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:42.400 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T18:20:42.414 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T18:20:42.458 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:47.892 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T18:20:47.910 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T18:20:47.941 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:47.965 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T18:20:47.983 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T18:20:48.022 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:48.244 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T18:20:48.258 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T18:20:48.302 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:48.319 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T18:20:48.337 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T18:20:48.341 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T18:20:48.356 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T18:20:48.393 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:48.400 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:48.689 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package thermald. 2022-02-21T18:20:48.702 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T18:20:48.703 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T18:20:48.716 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T18:20:48.747 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:48.760 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:48.794 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package thermald. 2022-02-21T18:20:48.808 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T18:20:48.852 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:49.147 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package thermald. 2022-02-21T18:20:49.160 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T18:20:49.197 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:49.247 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package upower. 2022-02-21T18:20:49.263 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:49.316 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T18:20:49.349 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package upower. 2022-02-21T18:20:49.364 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:49.417 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T18:20:49.722 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T18:20:49.737 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T18:20:49.755 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package upower. 2022-02-21T18:20:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T18:20:49.780 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:49.823 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T18:20:49.831 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T18:20:49.844 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T18:20:49.889 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:50.074 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:50.175 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:50.221 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:50.263 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T18:20:50.278 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T18:20:50.315 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:50.322 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:50.347 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:50.440 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:50.457 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:50.541 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:50.574 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T18:20:50.667 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:50.702 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:50.752 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T18:20:50.793 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:50.987 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T18:20:50.988 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T18:20:50.988 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T18:20:50.989 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T18:20:51.138 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T18:20:51.266 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T18:20:51.287 INFO:teuthology.orchestra.run.smithi179.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:51.391 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T18:20:51.398 INFO:teuthology.orchestra.run.smithi140.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:51.446 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:51.470 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:51.624 INFO:teuthology.orchestra.run.smithi140.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T18:20:51.634 INFO:teuthology.orchestra.run.smithi140.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T18:20:51.634 INFO:teuthology.orchestra.run.smithi140.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T18:20:51.647 INFO:teuthology.orchestra.run.smithi179.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T18:20:51.661 INFO:teuthology.orchestra.run.smithi179.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T18:20:51.661 INFO:teuthology.orchestra.run.smithi179.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T18:20:51.831 INFO:teuthology.orchestra.run.smithi081.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:51.884 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T18:20:52.055 INFO:teuthology.orchestra.run.smithi081.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T18:20:52.063 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T18:20:52.063 INFO:teuthology.orchestra.run.smithi081.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T18:20:52.113 INFO:teuthology.orchestra.run.smithi140.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T18:20:52.155 INFO:teuthology.orchestra.run.smithi179.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T18:20:52.346 INFO:teuthology.orchestra.run.smithi140.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:52.398 INFO:teuthology.orchestra.run.smithi179.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:52.405 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:52.442 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:52.543 INFO:teuthology.orchestra.run.smithi081.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T18:20:52.710 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:52.711 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:52.761 INFO:teuthology.orchestra.run.smithi081.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T18:20:52.768 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:52.769 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:52.805 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T18:20:53.129 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:53.129 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T18:20:53.197 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:53.206 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:53.620 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:54.462 INFO:teuthology.orchestra.run.smithi179.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T18:20:54.462 INFO:teuthology.orchestra.run.smithi179.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T18:20:54.465 INFO:teuthology.orchestra.run.smithi140.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T18:20:54.465 INFO:teuthology.orchestra.run.smithi140.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T18:20:54.626 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:54.648 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:54.860 INFO:teuthology.orchestra.run.smithi081.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T18:20:54.860 INFO:teuthology.orchestra.run.smithi081.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T18:20:55.029 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:55.927 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:56.317 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:56.389 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:20:57.596 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:57.660 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:57.724 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T18:20:57.731 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T18:20:57.802 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T18:20:57.850 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T18:21:01.612 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:21:02.273 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T18:21:02.557 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:21:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:21:03.031 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:21:03.186 INFO:teuthology.orchestra.run.smithi179.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T18:21:03.186 INFO:teuthology.orchestra.run.smithi179.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:03.201 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T18:21:03.310 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:21:03.502 INFO:teuthology.orchestra.run.smithi140.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T18:21:03.502 INFO:teuthology.orchestra.run.smithi140.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:03.691 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T18:21:03.783 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T18:21:03.964 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T18:21:03.964 INFO:teuthology.orchestra.run.smithi081.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:26.829 INFO:teuthology.orchestra.run.smithi179.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T18:21:26.829 INFO:teuthology.orchestra.run.smithi179.stdout:Sourcing file `/etc/default/grub' 2022-02-21T18:21:26.846 INFO:teuthology.orchestra.run.smithi179.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:26.872 INFO:teuthology.orchestra.run.smithi179.stdout:Generating grub configuration file ... 2022-02-21T18:21:27.061 INFO:teuthology.orchestra.run.smithi140.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T18:21:27.062 INFO:teuthology.orchestra.run.smithi140.stdout:Sourcing file `/etc/default/grub' 2022-02-21T18:21:27.074 INFO:teuthology.orchestra.run.smithi140.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:27.098 INFO:teuthology.orchestra.run.smithi140.stdout:Generating grub configuration file ... 2022-02-21T18:21:27.228 INFO:teuthology.orchestra.run.smithi179.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:27.259 INFO:teuthology.orchestra.run.smithi179.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:27.430 INFO:teuthology.orchestra.run.smithi140.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:27.431 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T18:21:27.431 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub' 2022-02-21T18:21:27.442 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:27.464 INFO:teuthology.orchestra.run.smithi140.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:27.510 INFO:teuthology.orchestra.run.smithi081.stdout:Generating grub configuration file ... 2022-02-21T18:21:27.634 INFO:teuthology.orchestra.run.smithi179.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:27.646 INFO:teuthology.orchestra.run.smithi179.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:27.826 INFO:teuthology.orchestra.run.smithi140.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:27.837 INFO:teuthology.orchestra.run.smithi140.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:27.873 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:27.907 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:28.377 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:28.388 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:29.418 INFO:teuthology.orchestra.run.smithi179.stdout:done 2022-02-21T18:21:29.541 INFO:teuthology.orchestra.run.smithi140.stdout:done 2022-02-21T18:21:29.814 DEBUG:teuthology.orchestra.run.smithi179:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:29.855 INFO:teuthology.orchestra.run.smithi179.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:29.856 INFO:teuthology.orchestra.run.smithi179.stdout:Status: install ok installed 2022-02-21T18:21:29.856 INFO:teuthology.orchestra.run.smithi179.stdout:Priority: optional 2022-02-21T18:21:29.856 INFO:teuthology.orchestra.run.smithi179.stdout:Section: kernel 2022-02-21T18:21:29.857 INFO:teuthology.orchestra.run.smithi179.stdout:Installed-Size: 19 2022-02-21T18:21:29.857 INFO:teuthology.orchestra.run.smithi179.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:29.857 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: amd64 2022-02-21T18:21:29.857 INFO:teuthology.orchestra.run.smithi179.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:29.858 INFO:teuthology.orchestra.run.smithi179.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:29.858 INFO:teuthology.orchestra.run.smithi179.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:29.858 INFO:teuthology.orchestra.run.smithi179.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:29.858 INFO:teuthology.orchestra.run.smithi179.stdout:Recommends: thermald 2022-02-21T18:21:29.859 INFO:teuthology.orchestra.run.smithi179.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:29.859 INFO:teuthology.orchestra.run.smithi179.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:29.859 INFO:teuthology.orchestra.run.smithi179.stdout: available. 2022-02-21T18:21:29.859 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T18:21:29.860 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T18:21:29.860 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T18:21:29.860 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-02-21T18:21:29.860 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T18:21:29.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get clean 2022-02-21T18:21:29.902 DEBUG:teuthology.orchestra.run.smithi140:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:29.928 INFO:teuthology.orchestra.run.smithi140.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:29.929 INFO:teuthology.orchestra.run.smithi140.stdout:Status: install ok installed 2022-02-21T18:21:29.929 INFO:teuthology.orchestra.run.smithi140.stdout:Priority: optional 2022-02-21T18:21:29.930 INFO:teuthology.orchestra.run.smithi140.stdout:Section: kernel 2022-02-21T18:21:29.930 INFO:teuthology.orchestra.run.smithi140.stdout:Installed-Size: 19 2022-02-21T18:21:29.930 INFO:teuthology.orchestra.run.smithi140.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:29.931 INFO:teuthology.orchestra.run.smithi140.stdout:Architecture: amd64 2022-02-21T18:21:29.931 INFO:teuthology.orchestra.run.smithi140.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:29.932 INFO:teuthology.orchestra.run.smithi140.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:29.932 INFO:teuthology.orchestra.run.smithi140.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:29.932 INFO:teuthology.orchestra.run.smithi140.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:29.933 INFO:teuthology.orchestra.run.smithi140.stdout:Recommends: thermald 2022-02-21T18:21:29.933 INFO:teuthology.orchestra.run.smithi140.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:29.935 INFO:teuthology.orchestra.run.smithi140.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:29.936 INFO:teuthology.orchestra.run.smithi140.stdout: available. 2022-02-21T18:21:29.937 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T18:21:29.937 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T18:21:29.938 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T18:21:29.938 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-02-21T18:21:29.938 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi140.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T18:21:29.939 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get clean 2022-02-21T18:21:29.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2022-02-21T18:21:30.014 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get update 2022-02-21T18:21:30.209 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:21:30.297 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:21:30.298 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:21:30.314 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:21:30.389 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:21:30.405 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:21:30.484 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:21:30.505 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:21:31.051 INFO:teuthology.orchestra.run.smithi081.stdout:done 2022-02-21T18:21:31.423 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:31.445 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:31.446 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2022-02-21T18:21:31.446 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2022-02-21T18:21:31.446 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2022-02-21T18:21:31.447 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 19 2022-02-21T18:21:31.447 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:31.447 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2022-02-21T18:21:31.448 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:31.448 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:31.448 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:31.448 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:31.448 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2022-02-21T18:21:31.449 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:31.449 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:31.449 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2022-02-21T18:21:31.449 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T18:21:31.450 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T18:21:31.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T18:21:31.450 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-02-21T18:21:31.450 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T18:21:31.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2022-02-21T18:21:31.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2022-02-21T18:21:31.728 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:21:31.753 DEBUG:teuthology.orchestra.run.smithi140:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:21:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:21:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:21:31.821 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:21:31.842 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:21:31.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:21:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:21:31.988 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:21:31.989 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:21:31.990 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:21:32.009 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:21:32.158 INFO:teuthology.orchestra.run.smithi140.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T18:21:32.159 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:21:32.160 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:21:32.161 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:21:32.161 DEBUG:teuthology.orchestra.run.smithi140:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:32.180 INFO:teuthology.orchestra.run.smithi140.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:32.181 INFO:teuthology.orchestra.run.smithi140.stdout:Status: install ok installed 2022-02-21T18:21:32.181 INFO:teuthology.orchestra.run.smithi140.stdout:Priority: optional 2022-02-21T18:21:32.181 INFO:teuthology.orchestra.run.smithi140.stdout:Section: kernel 2022-02-21T18:21:32.181 INFO:teuthology.orchestra.run.smithi140.stdout:Installed-Size: 19 2022-02-21T18:21:32.182 INFO:teuthology.orchestra.run.smithi140.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:32.182 INFO:teuthology.orchestra.run.smithi140.stdout:Architecture: amd64 2022-02-21T18:21:32.183 INFO:teuthology.orchestra.run.smithi140.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:32.183 INFO:teuthology.orchestra.run.smithi140.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:32.183 INFO:teuthology.orchestra.run.smithi140.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:32.183 INFO:teuthology.orchestra.run.smithi140.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:32.184 INFO:teuthology.orchestra.run.smithi140.stdout:Recommends: thermald 2022-02-21T18:21:32.184 INFO:teuthology.orchestra.run.smithi140.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:32.185 INFO:teuthology.orchestra.run.smithi140.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:32.185 INFO:teuthology.orchestra.run.smithi140.stdout: available. 2022-02-21T18:21:32.185 DEBUG:teuthology.orchestra.run.smithi140:> mktemp 2022-02-21T18:21:32.190 INFO:teuthology.orchestra.run.smithi140.stdout:/tmp/tmp.dipRwhgokO 2022-02-21T18:21:32.191 DEBUG:teuthology.orchestra.run.smithi140:> sudo cp /boot/grub/grub.cfg /tmp/tmp.dipRwhgokO 2022-02-21T18:21:32.248 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0666 /tmp/tmp.dipRwhgokO 2022-02-21T18:21:32.330 DEBUG:teuthology.orchestra.remote:smithi140:/tmp/tmp.dipRwhgokO is 8KB 2022-02-21T18:21:32.343 INFO:teuthology.orchestra.run.smithi179.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T18:21:32.343 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:21:32.345 DEBUG:teuthology.orchestra.run.smithi179:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:32.350 DEBUG:teuthology.orchestra.run.smithi140:> rm -fr /tmp/tmp.dipRwhgokO 2022-02-21T18:21:32.358 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.368 INFO:teuthology.orchestra.run.smithi179.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:32.369 INFO:teuthology.orchestra.run.smithi179.stdout:Status: install ok installed 2022-02-21T18:21:32.369 INFO:teuthology.orchestra.run.smithi179.stdout:Priority: optional 2022-02-21T18:21:32.369 INFO:teuthology.orchestra.run.smithi179.stdout:Section: kernel 2022-02-21T18:21:32.369 INFO:teuthology.orchestra.run.smithi179.stdout:Installed-Size: 19 2022-02-21T18:21:32.370 INFO:teuthology.orchestra.run.smithi179.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:32.370 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: amd64 2022-02-21T18:21:32.370 INFO:teuthology.orchestra.run.smithi179.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:32.373 INFO:teuthology.orchestra.run.smithi179.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:32.373 INFO:teuthology.orchestra.run.smithi179.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:32.373 INFO:teuthology.orchestra.run.smithi179.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:32.373 INFO:teuthology.orchestra.run.smithi179.stdout:Recommends: thermald 2022-02-21T18:21:32.374 INFO:teuthology.orchestra.run.smithi179.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:32.374 INFO:teuthology.orchestra.run.smithi179.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:32.374 INFO:teuthology.orchestra.run.smithi179.stdout: available. 2022-02-21T18:21:32.375 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2022-02-21T18:21:32.380 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.Z994fm1q5x 2022-02-21T18:21:32.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Z994fm1q5x 2022-02-21T18:21:32.415 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:21:32.416 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.416 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.Z994fm1q5x 2022-02-21T18:21:32.453 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T18:21:32.453 DEBUG:teuthology.orchestra.run.smithi140:> sudo update-grub 2022-02-21T18:21:32.481 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.Z994fm1q5x is 8KB 2022-02-21T18:21:32.494 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.Z994fm1q5x 2022-02-21T18:21:32.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.560 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:21:32.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:32.593 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T18:21:32.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo update-grub 2022-02-21T18:21:33.362 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:21:33.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T18:21:33.452 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:21:33.625 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:21:33.626 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:21:33.640 INFO:teuthology.orchestra.run.smithi140.stderr:Sourcing file `/etc/default/grub' 2022-02-21T18:21:33.642 INFO:teuthology.orchestra.run.smithi140.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:33.655 INFO:teuthology.orchestra.run.smithi140.stderr:Generating grub configuration file ... 2022-02-21T18:21:33.804 INFO:teuthology.orchestra.run.smithi081.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T18:21:33.804 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T18:21:33.806 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T18:21:33.829 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T18:21:33.829 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2022-02-21T18:21:33.830 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2022-02-21T18:21:33.830 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2022-02-21T18:21:33.830 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 19 2022-02-21T18:21:33.830 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T18:21:33.831 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2022-02-21T18:21:33.831 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta-hwe-5.13 2022-02-21T18:21:33.831 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T18:21:33.831 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T18:21:33.832 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T18:21:33.832 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2022-02-21T18:21:33.832 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2022-02-21T18:21:33.832 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2022-02-21T18:21:33.833 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2022-02-21T18:21:33.833 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2022-02-21T18:21:33.834 INFO:teuthology.orchestra.run.smithi179.stderr:Sourcing file `/etc/default/grub' 2022-02-21T18:21:33.834 INFO:teuthology.orchestra.run.smithi179.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:33.841 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.IL1Uw1POY1 2022-02-21T18:21:33.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IL1Uw1POY1 2022-02-21T18:21:33.853 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2022-02-21T18:21:33.880 INFO:teuthology.orchestra.run.smithi140.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:33.892 INFO:teuthology.orchestra.run.smithi140.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:33.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.IL1Uw1POY1 2022-02-21T18:21:33.936 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.IL1Uw1POY1 is 8KB 2022-02-21T18:21:33.949 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.IL1Uw1POY1 2022-02-21T18:21:33.956 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:34.011 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:21:34.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T18:21:34.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T18:21:34.044 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T18:21:34.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo update-grub 2022-02-21T18:21:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:34.123 INFO:teuthology.orchestra.run.smithi179.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:34.266 INFO:teuthology.orchestra.run.smithi140.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:34.278 INFO:teuthology.orchestra.run.smithi140.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:34.522 INFO:teuthology.orchestra.run.smithi179.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:34.532 INFO:teuthology.orchestra.run.smithi179.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:34.987 INFO:teuthology.orchestra.run.smithi140.stderr:done 2022-02-21T18:21:34.988 DEBUG:teuthology.orchestra.run.smithi140:> sudo shutdown -r now 2022-02-21T18:21:35.240 INFO:teuthology.orchestra.run.smithi179.stderr:done 2022-02-21T18:21:35.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2022-02-21T18:21:35.292 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub' 2022-02-21T18:21:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T18:21:35.357 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2022-02-21T18:21:35.636 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T18:21:35.648 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T18:21:36.094 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T18:21:36.105 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T18:21:37.730 INFO:teuthology.orchestra.run.smithi081.stderr:done 2022-02-21T18:21:37.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2022-02-21T18:22:05.018 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-02-21T18:22:05.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2022-02-21T18:22:05.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:05.244 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-02-21T18:22:05.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-02-21T18:22:05.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:07.737 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-02-21T18:22:07.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-02-21T18:22:07.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:23.435 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-02-21T18:22:23.691 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2022-02-21T18:22:26.251 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2022-02-21T18:22:53.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2022-02-21T18:22:53.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:53.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-02-21T18:22:53.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:56.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-02-21T18:22:56.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:22:56.747 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2022-02-21T18:22:59.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2022-02-21T18:23:25.110 DEBUG:teuthology.orchestra.run.smithi140:> true 2022-02-21T18:23:25.633 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi140.front.sepia.ceph.com' 2022-02-21T18:23:25.633 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-02-21T18:23:25.634 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2022-02-21T18:23:25.639 INFO:teuthology.orchestra.run.smithi140.stdout:5.13.0-30-generic 2022-02-21T18:23:25.639 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T18:23:25.640 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T18:23:25.640 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T18:23:26.641 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-02-21T18:23:26.642 DEBUG:teuthology.orchestra.run.smithi140:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T18:23:26.712 INFO:teuthology.orchestra.run.smithi140.stdout:ttyS1 2022-02-21T18:23:26.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-02-21T18:23:26.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:23:26.815 DEBUG:teuthology.parallel:result is None 2022-02-21T18:23:27.034 DEBUG:teuthology.orchestra.run.smithi179:> true 2022-02-21T18:23:27.585 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-02-21T18:23:27.585 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-02-21T18:23:27.586 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2022-02-21T18:23:27.634 INFO:teuthology.orchestra.run.smithi179.stdout:5.13.0-30-generic 2022-02-21T18:23:27.634 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T18:23:27.635 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T18:23:27.635 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T18:23:28.636 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-02-21T18:23:28.636 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T18:23:28.691 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2022-02-21T18:23:28.714 DEBUG:teuthology.parallel:result is None 2022-02-21T18:23:29.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-02-21T18:23:29.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:23:29.473 DEBUG:teuthology.orchestra.run.smithi081:> true 2022-02-21T18:23:30.005 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-02-21T18:23:30.006 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-02-21T18:23:30.006 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2022-02-21T18:23:30.055 INFO:teuthology.orchestra.run.smithi081.stdout:5.13.0-30-generic 2022-02-21T18:23:30.055 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T18:23:30.055 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T18:23:30.056 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T18:23:31.056 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-02-21T18:23:31.057 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T18:23:31.138 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2022-02-21T18:23:31.163 DEBUG:teuthology.parallel:result is None 2022-02-21T18:23:31.164 INFO:teuthology.run_tasks:Running task internal.base... 2022-02-21T18:23:31.175 INFO:teuthology.task.internal:Creating test directory... 2022-02-21T18:23:31.176 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T18:23:31.178 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T18:23:31.181 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T18:23:31.189 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-02-21T18:23:31.225 INFO:teuthology.run_tasks:Running task internal.archive... 2022-02-21T18:23:31.258 INFO:teuthology.task.internal:Creating archive directory... 2022-02-21T18:23:31.259 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T18:23:31.262 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T18:23:31.265 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T18:23:31.294 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-02-21T18:23:31.307 INFO:teuthology.task.internal:Enabling coredump saving... 2022-02-21T18:23:31.308 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T18:23:31.311 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T18:23:31.329 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T18:23:31.332 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.341 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.353 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.359 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.365 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.370 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T18:23:31.371 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-02-21T18:23:31.382 INFO:teuthology.task.internal:Configuring sudo... 2022-02-21T18:23:31.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T18:23:31.386 DEBUG:teuthology.orchestra.run.smithi140:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T18:23:31.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T18:23:31.431 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-02-21T18:23:31.444 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-02-21T18:23:31.444 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T18:23:31.447 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T18:23:31.450 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T18:23:31.457 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T18:23:31.463 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T18:23:31.512 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:23:31.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T18:23:31.537 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T18:23:31.546 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T18:23:31.596 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:23:31.597 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T18:23:31.620 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T18:23:31.629 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T18:23:31.636 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:23:31.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T18:23:31.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2022-02-21T18:23:31.669 DEBUG:teuthology.orchestra.run.smithi140:> sudo service rsyslog restart 2022-02-21T18:23:31.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2022-02-21T18:23:31.753 INFO:teuthology.run_tasks:Running task internal.timer... 2022-02-21T18:23:31.762 INFO:teuthology.task.internal:Starting timer... 2022-02-21T18:23:31.763 INFO:teuthology.run_tasks:Running task pcp... 2022-02-21T18:23:31.837 INFO:teuthology.run_tasks:Running task selinux... 2022-02-21T18:23:31.925 DEBUG:teuthology.task.selinux:Excluding smithi081: OS 'ubuntu' does not support SELinux 2022-02-21T18:23:31.925 DEBUG:teuthology.task.selinux:Excluding smithi140: OS 'ubuntu' does not support SELinux 2022-02-21T18:23:31.926 DEBUG:teuthology.task.selinux:Excluding smithi179: OS 'ubuntu' does not support SELinux 2022-02-21T18:23:31.926 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-02-21T18:23:31.927 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-02-21T18:23:31.927 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-02-21T18:23:31.927 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-02-21T18:23:31.961 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-02-21T18:23:31.967 INFO:teuthology.repo_utils:Fetching master from origin 2022-02-21T18:23:31.999 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-02-21T18:23:32.021 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-02-21T18:23:32.023 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi140.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-02-21T18:26:36.256 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi140.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2022-02-21T18:26:36.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-02-21T18:26:36.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:26:36.302 DEBUG:teuthology.orchestra.run.smithi081:> true 2022-02-21T18:26:36.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2022-02-21T18:26:36.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:26:36.469 DEBUG:teuthology.orchestra.run.smithi140:> true 2022-02-21T18:26:36.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-02-21T18:26:36.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T18:26:36.632 DEBUG:teuthology.orchestra.run.smithi179:> true 2022-02-21T18:26:36.748 INFO:teuthology.run_tasks:Running task clock... 2022-02-21T18:26:36.762 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-02-21T18:26:36.763 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T18:26:36.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T18:26:36.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T18:26:36.770 DEBUG:teuthology.orchestra.run.smithi140:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T18:26:36.773 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T18:26:36.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T18:26:36.800 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T18:26:36.801 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Command line: ntpd -gq 2022-02-21T18:26:36.801 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: proto: precision = 0.065 usec (-24) 2022-02-21T18:26:36.802 INFO:teuthology.orchestra.run.smithi081.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.802 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.803 INFO:teuthology.orchestra.run.smithi081.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.805 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.805 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T18:26:36.806 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T18:26:36.806 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T18:26:36.806 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen normally on 3 enp3s0f1 172.21.15.81:123 2022-02-21T18:26:36.806 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen normally on 4 lo [::1]:123 2022-02-21T18:26:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d753%5]:123 2022-02-21T18:26:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:36 ntpd[8967]: Listening on routing socket on fd #22 for interface updates 2022-02-21T18:26:36.808 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T18:26:36.808 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Command line: ntpd -gq 2022-02-21T18:26:36.809 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: proto: precision = 0.078 usec (-24) 2022-02-21T18:26:36.809 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T18:26:36.809 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Command line: ntpd -gq 2022-02-21T18:26:36.810 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: proto: precision = 0.096 usec (-23) 2022-02-21T18:26:36.810 INFO:teuthology.orchestra.run.smithi179.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.810 INFO:teuthology.orchestra.run.smithi140.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.811 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.812 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T18:26:36.812 INFO:teuthology.orchestra.run.smithi179.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.812 INFO:teuthology.orchestra.run.smithi140.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.814 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.815 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T18:26:36.815 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T18:26:36.815 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T18:26:36.815 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen normally on 3 enp3s0f1 172.21.15.179:123 2022-02-21T18:26:36.816 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen normally on 4 lo [::1]:123 2022-02-21T18:26:36.816 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:725b%5]:123 2022-02-21T18:26:36.816 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:36 ntpd[8940]: Listening on routing socket on fd #22 for interface updates 2022-02-21T18:26:36.817 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T18:26:36.817 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T18:26:36.817 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T18:26:36.817 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T18:26:36.818 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen normally on 3 ens1f1 172.21.15.140:123 2022-02-21T18:26:36.818 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen normally on 4 lo [::1]:123 2022-02-21T18:26:36.818 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listen normally on 5 ens1f1 [fe80::ae1f:6bff:fea5:ca31%5]:123 2022-02-21T18:26:36.818 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:36 ntpd[8871]: Listening on routing socket on fd #22 for interface updates 2022-02-21T18:26:37.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.804 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.804 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.804 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.804 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.805 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.805 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.805 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.805 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.806 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:37 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.809 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.809 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.810 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.810 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.810 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.810 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.810 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.811 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.811 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.811 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.811 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.812 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:37 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.813 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.813 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.813 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.813 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.813 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.814 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.814 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.814 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.814 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.815 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.815 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.815 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.815 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.816 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:37.816 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:37.816 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:37.816 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:37.816 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:37 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:38.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:38.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:38.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:38 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:39 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:39 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:39.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:39.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:39.821 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:39.821 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:39 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:40.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:40.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:40.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:40.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:40.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:40.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:40.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:40.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:40.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:40.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:40.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:40.803 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:40 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:41 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.811 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.812 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:41 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.815 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.815 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.817 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.818 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.819 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:41.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:41.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:41.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:41.820 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:41 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:42.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:42.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:42.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:42.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:42.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:42.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:42.800 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:42.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:42.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:42.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:42.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:42.802 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:42 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:43.798 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.798 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.798 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.798 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:43.799 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:43.800 INFO:teuthology.orchestra.run.smithi081.stdout:21 Feb 18:26:43 ntpd[8967]: ntpd: time slew -0.000461 s 2022-02-21T18:26:43.800 INFO:teuthology.orchestra.run.smithi081.stdout:ntpd: time slew -0.000461s 2022-02-21T18:26:43.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-02-21T18:26:43.801 INFO:teuthology.orchestra.run.smithi081.stderr:21 Feb 18:26:43 ntpd[8967]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T18:26:43.806 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.806 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.806 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.807 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.808 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:43.809 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:43.809 INFO:teuthology.orchestra.run.smithi140.stdout:21 Feb 18:26:43 ntpd[8871]: ntpd: time slew -0.010125 s 2022-02-21T18:26:43.809 INFO:teuthology.orchestra.run.smithi140.stdout:ntpd: time slew -0.010125s 2022-02-21T18:26:43.810 INFO:teuthology.orchestra.run.smithi140.stderr:21 Feb 18:26:43 ntpd[8871]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T18:26:43.812 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.812 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.812 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.812 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.813 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.813 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.813 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.813 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.814 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T18:26:43.814 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T18:26:43.814 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T18:26:43.814 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T18:26:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:21 Feb 18:26:43 ntpd[8940]: ntpd: time slew -0.011936 s 2022-02-21T18:26:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:ntpd: time slew -0.011936s 2022-02-21T18:26:43.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-02-21T18:26:43.816 INFO:teuthology.orchestra.run.smithi179.stderr:21 Feb 18:26:43 ntpd[8940]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T18:26:43.862 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T18:26:43.862 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== 2022-02-21T18:26:43.863 INFO:teuthology.orchestra.run.smithi179.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.863 INFO:teuthology.orchestra.run.smithi179.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.863 INFO:teuthology.orchestra.run.smithi179.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.863 INFO:teuthology.orchestra.run.smithi179.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.865 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T18:26:43.865 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2022-02-21T18:26:43.866 INFO:teuthology.orchestra.run.smithi081.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.866 INFO:teuthology.orchestra.run.smithi081.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.866 INFO:teuthology.orchestra.run.smithi081.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.866 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.867 INFO:teuthology.orchestra.run.smithi140.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T18:26:43.868 INFO:teuthology.orchestra.run.smithi140.stdout:============================================================================== 2022-02-21T18:26:43.868 INFO:teuthology.orchestra.run.smithi140.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.868 INFO:teuthology.orchestra.run.smithi140.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.868 INFO:teuthology.orchestra.run.smithi140.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.869 INFO:teuthology.orchestra.run.smithi140.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T18:26:43.869 INFO:teuthology.run_tasks:Running task kubeadm... 2022-02-21T18:26:43.887 INFO:tasks.kubeadm:Kubeadm start 2022-02-21T18:26:43.888 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-02-21T18:26:43.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T18:26:43.891 DEBUG:teuthology.orchestra.run.smithi140:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T18:26:43.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T18:26:43.915 INFO:teuthology.orchestra.run.smithi140.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T18:26:43.916 INFO:teuthology.orchestra.run.smithi081.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T18:26:43.922 INFO:teuthology.orchestra.run.smithi140.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T18:26:43.925 INFO:teuthology.orchestra.run.smithi081.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T18:26:43.932 INFO:teuthology.orchestra.run.smithi140.stdout:net.ipv4.ip_forward = 1 2022-02-21T18:26:43.933 INFO:teuthology.orchestra.run.smithi179.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T18:26:43.934 INFO:teuthology.orchestra.run.smithi081.stdout:net.ipv4.ip_forward = 1 2022-02-21T18:26:43.939 INFO:teuthology.orchestra.run.smithi179.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T18:26:43.950 INFO:teuthology.orchestra.run.smithi179.stdout:net.ipv4.ip_forward = 1 2022-02-21T18:26:43.996 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:26:43.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T18:26:44.025 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:26:44.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/docker/daemon.json 2022-02-21T18:26:44.039 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:26:44.040 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T18:26:44.056 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:26:44.057 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/docker/daemon.json 2022-02-21T18:26:44.073 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:26:44.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T18:26:44.089 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:26:44.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/docker/daemon.json 2022-02-21T18:26:44.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl restart docker || true 2022-02-21T18:26:44.104 DEBUG:teuthology.orchestra.run.smithi140:> sudo systemctl restart docker || true 2022-02-21T18:26:44.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl restart docker || true 2022-02-21T18:26:45.308 INFO:tasks.kubeadm:Identifying pod subnet 2022-02-21T18:26:45.315 INFO:tasks.kubeadm:172.21.15.81 is in 172.21.0.0/20 at pos 3920 2022-02-21T18:26:45.329 INFO:tasks.kubeadm:Pod subnet: 10.250.128.0/21 2022-02-21T18:26:45.329 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-02-21T18:26:45.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T18:26:45.333 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T18:26:45.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T18:26:45.354 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:45.355 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:45.355 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:45.359 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:45.359 INFO:teuthology.orchestra.run.smithi140.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:45.359 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:45.360 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:45.360 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:45.361 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:45.487 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:45.488 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:45.601 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:45.602 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:45.613 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:45.613 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:45.683 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:45.684 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:45.684 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:45.766 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:45.767 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:45.767 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:46.957 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:26:46.972 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:26:47.094 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:26:47.126 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:26:47.127 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:26:47.145 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:26:47.146 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:26:47.220 INFO:teuthology.orchestra.run.smithi179.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:47.233 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:47.234 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:47.234 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:47.236 INFO:teuthology.orchestra.run.smithi140.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:47.253 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:47.253 INFO:teuthology.orchestra.run.smithi140.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:47.253 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:47.268 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:26:47.268 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:26:47.286 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:26:47.308 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:26:47.362 INFO:teuthology.orchestra.run.smithi081.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:47.378 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:47.378 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:47.379 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:47.435 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:26:47.458 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:26:47.459 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:26:47.482 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:26:47.483 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:26:47.609 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:26:47.610 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:26:47.623 INFO:teuthology.orchestra.run.smithi179.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T18:26:47.624 INFO:teuthology.orchestra.run.smithi179.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T18:26:47.624 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2022-02-21T18:26:47.625 INFO:teuthology.orchestra.run.smithi179.stdout: apt-transport-https 2022-02-21T18:26:47.660 INFO:teuthology.orchestra.run.smithi140.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T18:26:47.660 INFO:teuthology.orchestra.run.smithi140.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T18:26:47.661 INFO:teuthology.orchestra.run.smithi140.stdout:The following NEW packages will be installed: 2022-02-21T18:26:47.661 INFO:teuthology.orchestra.run.smithi140.stdout: apt-transport-https 2022-02-21T18:26:47.787 INFO:teuthology.orchestra.run.smithi081.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T18:26:47.788 INFO:teuthology.orchestra.run.smithi081.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T18:26:47.788 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2022-02-21T18:26:47.789 INFO:teuthology.orchestra.run.smithi081.stdout: apt-transport-https 2022-02-21T18:26:47.827 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:47.828 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 4,680 B of archives. 2022-02-21T18:26:47.828 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T18:26:47.829 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T18:26:47.863 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:47.863 INFO:teuthology.orchestra.run.smithi140.stdout:Need to get 4,680 B of archives. 2022-02-21T18:26:47.864 INFO:teuthology.orchestra.run.smithi140.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T18:26:47.865 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T18:26:47.977 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:47.977 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 4,680 B of archives. 2022-02-21T18:26:47.978 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T18:26:47.978 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T18:26:48.053 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:48.053 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:48.054 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:48.081 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:48.081 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:48.082 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:48.086 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:48.087 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:48.087 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:48.087 INFO:teuthology.orchestra.run.smithi179.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:48.109 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:48.110 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:48.110 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:48.114 INFO:teuthology.orchestra.run.smithi140.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:48.137 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 4,680 B in 0s (23.7 kB/s) 2022-02-21T18:26:48.162 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 4,680 B in 0s (24.1 kB/s) 2022-02-21T18:26:48.213 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T18:26:48.219 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:48.219 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:48.219 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:48.243 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:48.248 INFO:teuthology.orchestra.run.smithi081.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:48.255 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T18:26:48.258 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T18:26:48.278 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T18:26:48.295 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T18:26:48.299 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 4,680 B in 0s (25.8 kB/s) 2022-02-21T18:26:48.323 INFO:teuthology.orchestra.run.smithi140.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T18:26:48.327 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T18:26:48.369 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T18:26:48.411 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T18:26:48.455 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T18:26:48.458 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T18:26:48.501 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T18:26:48.599 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T18:26:48.662 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T18:26:48.804 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T18:26:49.247 INFO:teuthology.orchestra.run.smithi140.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T18:26:49.252 INFO:teuthology.orchestra.run.smithi179.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T18:26:49.262 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:49.262 INFO:teuthology.orchestra.run.smithi140.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:49.262 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:49.266 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:49.266 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:49.267 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:49.346 INFO:teuthology.orchestra.run.smithi081.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T18:26:49.360 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:49.360 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:49.360 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:49.550 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:49.551 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:49.553 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:49.553 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:49.641 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:49.646 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:49.654 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T18:26:49.654 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T18:26:49.746 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T18:26:49.747 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:49.749 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:49.853 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T18:26:52.195 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T18:26:52.317 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T18:26:52.385 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T18:26:52.469 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 63.3 kB in 3s (20.7 kB/s) 2022-02-21T18:26:52.518 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T18:26:52.663 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 63.3 kB in 3s (20.0 kB/s) 2022-02-21T18:26:53.311 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:26:53.478 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:26:53.479 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:26:53.493 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:26:53.568 INFO:teuthology.orchestra.run.smithi140.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:53.585 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:53.585 INFO:teuthology.orchestra.run.smithi140.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:53.585 INFO:teuthology.orchestra.run.smithi140.stderr: 2022-02-21T18:26:53.638 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2022-02-21T18:26:53.668 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:26:53.668 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:26:53.762 INFO:teuthology.orchestra.run.smithi081.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:53.775 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:53.775 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:53.775 INFO:teuthology.orchestra.run.smithi081.stderr: 2022-02-21T18:26:53.814 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2022-02-21T18:26:53.815 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2022-02-21T18:26:53.832 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2022-02-21T18:26:53.957 INFO:teuthology.orchestra.run.smithi140.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T18:26:53.957 INFO:teuthology.orchestra.run.smithi140.stdout:bridge-utils set to manually installed. 2022-02-21T18:26:53.957 INFO:teuthology.orchestra.run.smithi140.stdout:The following additional packages will be installed: 2022-02-21T18:26:53.958 INFO:teuthology.orchestra.run.smithi140.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T18:26:53.958 INFO:teuthology.orchestra.run.smithi140.stdout:Suggested packages: 2022-02-21T18:26:53.959 INFO:teuthology.orchestra.run.smithi140.stdout: nftables 2022-02-21T18:26:53.999 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2022-02-21T18:26:54.000 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2022-02-21T18:26:54.008 INFO:teuthology.orchestra.run.smithi140.stdout:The following NEW packages will be installed: 2022-02-21T18:26:54.009 INFO:teuthology.orchestra.run.smithi140.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T18:26:54.129 INFO:teuthology.orchestra.run.smithi081.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T18:26:54.129 INFO:teuthology.orchestra.run.smithi081.stdout:bridge-utils set to manually installed. 2022-02-21T18:26:54.130 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2022-02-21T18:26:54.130 INFO:teuthology.orchestra.run.smithi081.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T18:26:54.130 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2022-02-21T18:26:54.131 INFO:teuthology.orchestra.run.smithi081.stdout: nftables 2022-02-21T18:26:54.177 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2022-02-21T18:26:54.178 INFO:teuthology.orchestra.run.smithi081.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T18:26:54.200 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:54.200 INFO:teuthology.orchestra.run.smithi140.stdout:Need to get 73.6 MB of archives. 2022-02-21T18:26:54.201 INFO:teuthology.orchestra.run.smithi140.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T18:26:54.201 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T18:26:54.251 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T18:26:54.366 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T18:26:54.367 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T18:26:54.368 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:54.368 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 73.6 MB of archives. 2022-02-21T18:26:54.369 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T18:26:54.369 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T18:26:54.422 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T18:26:54.482 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T18:26:54.485 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T18:26:54.550 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T18:26:54.566 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T18:26:54.614 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 63.3 kB in 5s (12.2 kB/s) 2022-02-21T18:26:54.655 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T18:26:54.706 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T18:26:55.206 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T18:26:55.231 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T18:26:55.438 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:26:55.615 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:26:55.616 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:26:55.705 INFO:teuthology.orchestra.run.smithi179.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T18:26:55.719 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:55.719 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T18:26:55.720 INFO:teuthology.orchestra.run.smithi179.stderr: 2022-02-21T18:26:55.750 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T18:26:55.757 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T18:26:55.774 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-02-21T18:26:55.947 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-02-21T18:26:55.949 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-02-21T18:26:55.976 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T18:26:56.008 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T18:26:56.092 INFO:teuthology.orchestra.run.smithi179.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T18:26:56.093 INFO:teuthology.orchestra.run.smithi179.stdout:bridge-utils set to manually installed. 2022-02-21T18:26:56.093 INFO:teuthology.orchestra.run.smithi179.stdout:The following additional packages will be installed: 2022-02-21T18:26:56.093 INFO:teuthology.orchestra.run.smithi179.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T18:26:56.094 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2022-02-21T18:26:56.094 INFO:teuthology.orchestra.run.smithi179.stdout: nftables 2022-02-21T18:26:56.140 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2022-02-21T18:26:56.141 INFO:teuthology.orchestra.run.smithi179.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T18:26:56.334 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T18:26:56.334 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 73.6 MB of archives. 2022-02-21T18:26:56.334 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T18:26:56.335 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T18:26:56.395 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T18:26:56.415 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:56.416 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:56.416 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:56.418 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:56.418 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:56.419 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:56.423 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:56.426 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:56.428 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:56.428 INFO:teuthology.orchestra.run.smithi140.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:56.433 INFO:teuthology.orchestra.run.smithi140.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:56.478 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 73.6 MB in 2s (36.7 MB/s) 2022-02-21T18:26:56.484 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 73.6 MB in 2s (33.8 MB/s) 2022-02-21T18:26:56.505 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T18:26:56.566 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package conntrack. 2022-02-21T18:26:56.574 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package conntrack. 2022-02-21T18:26:56.609 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T18:26:56.612 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T18:26:56.618 INFO:teuthology.orchestra.run.smithi140.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T18:26:56.621 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T18:26:56.657 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T18:26:56.658 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T18:26:56.665 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T18:26:56.824 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T18:26:56.976 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cri-tools. 2022-02-21T18:26:56.990 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T18:26:56.992 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package cri-tools. 2022-02-21T18:26:57.005 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T18:26:57.035 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T18:26:57.043 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T18:26:57.573 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T18:26:58.012 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T18:26:58.131 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ebtables. 2022-02-21T18:26:58.137 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ebtables. 2022-02-21T18:26:58.143 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T18:26:58.151 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T18:26:58.202 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T18:26:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T18:26:58.212 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T18:26:58.465 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T18:26:58.481 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T18:26:58.499 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T18:26:58.513 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T18:26:58.531 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T18:26:58.557 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T18:26:58.587 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T18:26:58.587 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T18:26:58.588 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: falling back to frontend: Readline 2022-02-21T18:26:58.596 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T18:26:58.597 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T18:26:58.597 INFO:teuthology.orchestra.run.smithi179.stderr:debconf: falling back to frontend: Teletype 2022-02-21T18:26:58.601 INFO:teuthology.orchestra.run.smithi179.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T18:26:58.651 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 73.6 MB in 2s (33.2 MB/s) 2022-02-21T18:26:58.745 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package conntrack. 2022-02-21T18:26:58.787 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T18:26:58.790 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T18:26:58.834 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T18:26:59.153 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package cri-tools. 2022-02-21T18:26:59.168 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T18:26:59.212 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T18:26:59.984 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package socat. 2022-02-21T18:26:59.996 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T18:27:00.026 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package socat. 2022-02-21T18:27:00.038 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T18:27:00.069 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T18:27:00.076 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T18:27:00.323 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ebtables. 2022-02-21T18:27:00.335 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T18:27:00.389 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T18:27:00.470 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package kubelet. 2022-02-21T18:27:00.484 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T18:27:00.496 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package kubelet. 2022-02-21T18:27:00.510 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T18:27:00.521 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T18:27:00.554 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T18:27:00.659 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T18:27:00.671 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T18:27:00.709 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T18:27:02.136 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package socat. 2022-02-21T18:27:02.151 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T18:27:02.195 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T18:27:02.606 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package kubelet. 2022-02-21T18:27:02.620 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T18:27:02.664 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T18:27:02.973 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package kubectl. 2022-02-21T18:27:02.985 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T18:27:03.022 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T18:27:03.023 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package kubectl. 2022-02-21T18:27:03.037 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T18:27:03.073 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T18:27:04.200 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package kubeadm. 2022-02-21T18:27:04.212 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T18:27:04.251 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package kubeadm. 2022-02-21T18:27:04.264 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T18:27:04.266 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T18:27:04.326 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T18:27:05.192 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package kubectl. 2022-02-21T18:27:05.204 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T18:27:05.267 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T18:27:05.445 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T18:27:05.520 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T18:27:05.570 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T18:27:05.646 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T18:27:05.696 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T18:27:05.772 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T18:27:06.469 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package kubeadm. 2022-02-21T18:27:06.482 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T18:27:06.519 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T18:27:06.549 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T18:27:06.658 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T18:27:06.674 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T18:27:06.759 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T18:27:06.792 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T18:27:06.868 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T18:27:06.918 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T18:27:07.019 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T18:27:07.135 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T18:27:07.243 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T18:27:07.636 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T18:27:07.656 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T18:27:07.746 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T18:27:07.806 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T18:27:07.933 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T18:27:08.021 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:27:08.287 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:27:08.826 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T18:27:08.952 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T18:27:09.070 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T18:27:09.204 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T18:27:09.411 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T18:27:09.939 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T18:27:10.333 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T18:27:11.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T18:27:11.228 DEBUG:teuthology.orchestra.run.smithi140:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T18:27:11.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T18:27:23.532 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T18:27:23.750 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T18:27:25.298 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T18:27:27.435 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T18:27:29.305 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T18:27:30.100 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T18:27:30.200 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T18:27:31.655 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T18:27:33.105 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T18:27:34.216 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T18:27:35.869 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T18:27:36.024 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T18:27:36.963 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T18:27:37.512 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T18:27:38.787 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T18:27:42.340 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T18:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T18:27:44.889 INFO:teuthology.orchestra.run.smithi081.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T18:27:45.368 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T18:27:46.386 INFO:teuthology.orchestra.run.smithi179.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T18:27:47.727 INFO:teuthology.orchestra.run.smithi140.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T18:27:47.731 INFO:tasks.kubeadm:Token: abcdef.x9fc2c7otseftqo1 2022-02-21T18:27:47.732 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): '172.21.15.81', Remote(name='ubuntu@smithi140.front.sepia.ceph.com'): '172.21.15.140', Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): '172.21.15.179'} 2022-02-21T18:27:47.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo kubeadm init --node-name smithi081 --token abcdef.x9fc2c7otseftqo1 --pod-network-cidr 10.250.128.0/21 2022-02-21T18:27:47.968 INFO:teuthology.orchestra.run.smithi081.stdout:[init] Using Kubernetes version: v1.23.4 2022-02-21T18:27:47.968 INFO:teuthology.orchestra.run.smithi081.stdout:[preflight] Running pre-flight checks 2022-02-21T18:27:48.240 INFO:teuthology.orchestra.run.smithi081.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-02-21T18:27:48.240 INFO:teuthology.orchestra.run.smithi081.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-02-21T18:27:48.241 INFO:teuthology.orchestra.run.smithi081.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-02-21T18:27:48.547 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-02-21T18:27:48.745 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "ca" certificate and key 2022-02-21T18:27:49.458 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "apiserver" certificate and key 2022-02-21T18:27:49.459 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi081] and IPs [10.96.0.1 172.21.15.81] 2022-02-21T18:27:49.557 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-02-21T18:27:49.636 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-02-21T18:27:49.761 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-02-21T18:27:49.890 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "etcd/ca" certificate and key 2022-02-21T18:27:50.035 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "etcd/server" certificate and key 2022-02-21T18:27:50.036 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi081] and IPs [172.21.15.81 127.0.0.1 ::1] 2022-02-21T18:27:50.245 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "etcd/peer" certificate and key 2022-02-21T18:27:50.246 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi081] and IPs [172.21.15.81 127.0.0.1 ::1] 2022-02-21T18:27:50.353 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-02-21T18:27:50.484 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-02-21T18:27:50.617 INFO:teuthology.orchestra.run.smithi081.stdout:[certs] Generating "sa" key and public key 2022-02-21T18:27:50.618 INFO:teuthology.orchestra.run.smithi081.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-02-21T18:27:51.101 INFO:teuthology.orchestra.run.smithi081.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-02-21T18:27:51.238 INFO:teuthology.orchestra.run.smithi081.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-02-21T18:27:51.408 INFO:teuthology.orchestra.run.smithi081.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-02-21T18:27:51.725 INFO:teuthology.orchestra.run.smithi081.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-02-21T18:27:51.742 INFO:teuthology.orchestra.run.smithi081.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T18:27:51.743 INFO:teuthology.orchestra.run.smithi081.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T18:27:51.743 INFO:teuthology.orchestra.run.smithi081.stdout:[kubelet-start] Starting the kubelet 2022-02-21T18:27:51.945 INFO:teuthology.orchestra.run.smithi081.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-02-21T18:27:51.946 INFO:teuthology.orchestra.run.smithi081.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-02-21T18:27:51.949 INFO:teuthology.orchestra.run.smithi081.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-02-21T18:27:51.950 INFO:teuthology.orchestra.run.smithi081.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-02-21T18:27:51.951 INFO:teuthology.orchestra.run.smithi081.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-02-21T18:27:51.953 INFO:teuthology.orchestra.run.smithi081.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-02-21T18:28:09.457 INFO:teuthology.orchestra.run.smithi081.stdout:[apiclient] All control plane components are healthy after 17.504222 seconds 2022-02-21T18:28:09.458 INFO:teuthology.orchestra.run.smithi081.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-02-21T18:28:09.531 INFO:teuthology.orchestra.run.smithi081.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-02-21T18:28:09.531 INFO:teuthology.orchestra.run.smithi081.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-02-21T18:28:10.160 INFO:teuthology.orchestra.run.smithi081.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-02-21T18:28:10.160 INFO:teuthology.orchestra.run.smithi081.stdout:[mark-control-plane] Marking the node smithi081 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-02-21T18:28:10.161 INFO:teuthology.orchestra.run.smithi081.stdout:[mark-control-plane] Marking the node smithi081 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-02-21T18:28:10.690 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] Using token: abcdef.x9fc2c7otseftqo1 2022-02-21T18:28:10.690 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-02-21T18:28:10.747 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-02-21T18:28:10.764 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-02-21T18:28:10.771 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-02-21T18:28:10.783 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-02-21T18:28:10.788 INFO:teuthology.orchestra.run.smithi081.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-02-21T18:28:10.822 INFO:teuthology.orchestra.run.smithi081.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-02-21T18:28:11.361 INFO:teuthology.orchestra.run.smithi081.stdout:[addons] Applied essential addon: CoreDNS 2022-02-21T18:28:11.511 INFO:teuthology.orchestra.run.smithi081.stdout:[addons] Applied essential addon: kube-proxy 2022-02-21T18:28:11.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.515 INFO:teuthology.orchestra.run.smithi081.stdout:Your Kubernetes control-plane has initialized successfully! 2022-02-21T18:28:11.515 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.515 INFO:teuthology.orchestra.run.smithi081.stdout:To start using your cluster, you need to run the following as a regular user: 2022-02-21T18:28:11.516 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.516 INFO:teuthology.orchestra.run.smithi081.stdout: mkdir -p $HOME/.kube 2022-02-21T18:28:11.516 INFO:teuthology.orchestra.run.smithi081.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-02-21T18:28:11.516 INFO:teuthology.orchestra.run.smithi081.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-02-21T18:28:11.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.517 INFO:teuthology.orchestra.run.smithi081.stdout:Alternatively, if you are the root user, you can run: 2022-02-21T18:28:11.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.517 INFO:teuthology.orchestra.run.smithi081.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-02-21T18:28:11.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.518 INFO:teuthology.orchestra.run.smithi081.stdout:You should now deploy a pod network to the cluster. 2022-02-21T18:28:11.518 INFO:teuthology.orchestra.run.smithi081.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-02-21T18:28:11.518 INFO:teuthology.orchestra.run.smithi081.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-02-21T18:28:11.518 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.519 INFO:teuthology.orchestra.run.smithi081.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-02-21T18:28:11.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:28:11.519 INFO:teuthology.orchestra.run.smithi081.stdout:kubeadm join 172.21.15.81:6443 --token abcdef.x9fc2c7otseftqo1 \ 2022-02-21T18:28:11.520 INFO:teuthology.orchestra.run.smithi081.stdout: --discovery-token-ca-cert-hash sha256:168049bac1cc502d07d27a39ac6e12faabb4c3e2d3bd55dbe89e8621412b98b4 2022-02-21T18:28:11.521 DEBUG:teuthology.orchestra.run.smithi140:> sudo kubeadm join 172.21.15.81:6443 --node-name smithi140 --token abcdef.x9fc2c7otseftqo1 --discovery-token-unsafe-skip-ca-verification 2022-02-21T18:28:11.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo kubeadm join 172.21.15.81:6443 --node-name smithi179 --token abcdef.x9fc2c7otseftqo1 --discovery-token-unsafe-skip-ca-verification 2022-02-21T18:28:11.562 INFO:teuthology.orchestra.run.smithi140.stdout:[preflight] Running pre-flight checks 2022-02-21T18:28:11.567 INFO:teuthology.orchestra.run.smithi179.stdout:[preflight] Running pre-flight checks 2022-02-21T18:28:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:[preflight] Reading configuration from the cluster... 2022-02-21T18:28:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-02-21T18:28:24.195 INFO:teuthology.orchestra.run.smithi140.stdout:[preflight] Reading configuration from the cluster... 2022-02-21T18:28:24.195 INFO:teuthology.orchestra.run.smithi140.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-02-21T18:28:24.602 INFO:teuthology.orchestra.run.smithi140.stderr:W0221 18:28:24.618670 10472 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-02-21T18:28:24.603 INFO:teuthology.orchestra.run.smithi179.stderr:W0221 18:28:24.618985 10539 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-02-21T18:28:24.619 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T18:28:24.619 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T18:28:24.620 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-start] Starting the kubelet 2022-02-21T18:28:24.623 INFO:teuthology.orchestra.run.smithi179.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T18:28:24.623 INFO:teuthology.orchestra.run.smithi179.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T18:28:24.623 INFO:teuthology.orchestra.run.smithi179.stdout:[kubelet-start] Starting the kubelet 2022-02-21T18:28:24.829 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-02-21T18:28:24.846 INFO:teuthology.orchestra.run.smithi179.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-02-21T18:28:30.365 INFO:teuthology.orchestra.run.smithi140.stdout: 2022-02-21T18:28:30.366 INFO:teuthology.orchestra.run.smithi140.stdout:This node has joined the cluster: 2022-02-21T18:28:30.366 INFO:teuthology.orchestra.run.smithi140.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-02-21T18:28:30.366 INFO:teuthology.orchestra.run.smithi140.stdout:* The Kubelet was informed of the new secure connection details. 2022-02-21T18:28:30.366 INFO:teuthology.orchestra.run.smithi140.stdout: 2022-02-21T18:28:30.367 INFO:teuthology.orchestra.run.smithi140.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-02-21T18:28:30.367 INFO:teuthology.orchestra.run.smithi140.stdout: 2022-02-21T18:28:38.866 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-02-21T18:28:38.866 INFO:teuthology.orchestra.run.smithi179.stdout:This node has joined the cluster: 2022-02-21T18:28:38.866 INFO:teuthology.orchestra.run.smithi179.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-02-21T18:28:38.867 INFO:teuthology.orchestra.run.smithi179.stdout:* The Kubelet was informed of the new secure connection details. 2022-02-21T18:28:38.867 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-02-21T18:28:38.867 INFO:teuthology.orchestra.run.smithi179.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-02-21T18:28:38.868 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-02-21T18:28:38.872 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:38.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-02-21T18:28:38.888 INFO:tasks.kubeadm:Setting up kubectl 2022-02-21T18:28:38.888 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T18:28:38.945 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T18:28:38.964 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T18:28:38.982 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:38.982 DEBUG:teuthology.orchestra.run.smithi081:> dd of=.kube/config 2022-02-21T18:28:38.990 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:38.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/root/.kube/config 2022-02-21T18:28:39.056 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:39.057 DEBUG:teuthology.orchestra.run.smithi140:> dd of=.kube/config 2022-02-21T18:28:39.065 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:39.065 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/root/.kube/config 2022-02-21T18:28:39.078 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:39.078 DEBUG:teuthology.orchestra.run.smithi179:> dd of=.kube/config 2022-02-21T18:28:39.086 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:39.086 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/root/.kube/config 2022-02-21T18:28:39.101 DEBUG:teuthology.orchestra.run.smithi081:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-02-21T18:28:41.900 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-02-21T18:28:42.039 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-02-21T18:28:42.064 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-02-21T18:28:42.117 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-02-21T18:28:42.222 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-02-21T18:28:42.258 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-02-21T18:28:42.405 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-02-21T18:28:42.411 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-02-21T18:28:42.466 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-02-21T18:28:42.574 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-02-21T18:28:42.599 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-02-21T18:28:42.657 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-02-21T18:28:42.766 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-02-21T18:28:42.816 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-02-21T18:28:42.918 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-02-21T18:28:42.982 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-02-21T18:28:43.125 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-02-21T18:28:43.175 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-02-21T18:28:43.283 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-02-21T18:28:43.343 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-02-21T18:28:43.502 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-02-21T18:28:43.528 INFO:teuthology.orchestra.run.smithi081.stdout:namespace/tigera-operator created 2022-02-21T18:28:43.534 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-02-21T18:28:43.559 INFO:teuthology.orchestra.run.smithi081.stdout:podsecuritypolicy.policy/tigera-operator created 2022-02-21T18:28:43.590 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/tigera-operator created 2022-02-21T18:28:43.668 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-02-21T18:28:43.717 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-02-21T18:28:43.776 INFO:teuthology.orchestra.run.smithi081.stdout:deployment.apps/tigera-operator created 2022-02-21T18:28:43.783 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f - 2022-02-21T18:28:46.763 INFO:teuthology.orchestra.run.smithi081.stdout:installation.operator.tigera.io/default created 2022-02-21T18:28:46.770 INFO:tasks.kubeadm:Scanning for scratch devices 2022-02-21T18:28:46.770 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:46.771 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:46.778 INFO:tasks.kubeadm:Scratch devices on smithi081: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:46.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T18:28:46.805 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records in 2022-02-21T18:28:46.805 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-02-21T18:28:46.805 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136277 s, 769 MB/s 2022-02-21T18:28:46.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T18:28:46.833 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records in 2022-02-21T18:28:46.833 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-02-21T18:28:46.833 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0158671 s, 661 MB/s 2022-02-21T18:28:46.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T18:28:46.903 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records in 2022-02-21T18:28:46.903 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-02-21T18:28:46.903 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138675 s, 756 MB/s 2022-02-21T18:28:46.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T18:28:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records in 2022-02-21T18:28:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-02-21T18:28:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148245 s, 707 MB/s 2022-02-21T18:28:46.932 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:46.933 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:46.940 INFO:tasks.kubeadm:Scratch devices on smithi140: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:46.941 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T18:28:46.964 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records in 2022-02-21T18:28:46.964 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records out 2022-02-21T18:28:46.964 INFO:teuthology.orchestra.run.smithi140.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132405 s, 792 MB/s 2022-02-21T18:28:46.965 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T18:28:46.989 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records in 2022-02-21T18:28:46.989 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records out 2022-02-21T18:28:46.990 INFO:teuthology.orchestra.run.smithi140.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144691 s, 725 MB/s 2022-02-21T18:28:46.991 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T18:28:47.065 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records in 2022-02-21T18:28:47.065 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records out 2022-02-21T18:28:47.066 INFO:teuthology.orchestra.run.smithi140.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0167858 s, 625 MB/s 2022-02-21T18:28:47.067 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T18:28:47.093 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records in 2022-02-21T18:28:47.093 INFO:teuthology.orchestra.run.smithi140.stderr:10+0 records out 2022-02-21T18:28:47.094 INFO:teuthology.orchestra.run.smithi140.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157925 s, 664 MB/s 2022-02-21T18:28:47.095 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:47.095 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:47.103 INFO:tasks.kubeadm:Scratch devices on smithi179: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:47.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T18:28:47.132 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records in 2022-02-21T18:28:47.133 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2022-02-21T18:28:47.133 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0176025 s, 596 MB/s 2022-02-21T18:28:47.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T18:28:47.209 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records in 2022-02-21T18:28:47.209 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2022-02-21T18:28:47.209 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0174989 s, 599 MB/s 2022-02-21T18:28:47.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T18:28:47.241 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records in 2022-02-21T18:28:47.241 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2022-02-21T18:28:47.242 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0173776 s, 603 MB/s 2022-02-21T18:28:47.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T18:28:47.274 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records in 2022-02-21T18:28:47.275 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2022-02-21T18:28:47.275 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165919 s, 632 MB/s 2022-02-21T18:28:47.293 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-02-21T18:28:47.293 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi081-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi081 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi081-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi081 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi081-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi081 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi081-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi081 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi140-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi140 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi140-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi140 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi140-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi140 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi140-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi140 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi179-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi179 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi179-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi179 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi179-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi179 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi179-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi179 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-02-21T18:28:47.294 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f - 2022-02-21T18:28:48.204 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi081-lv-1 created 2022-02-21T18:28:48.226 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi081-lv-2 created 2022-02-21T18:28:48.234 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi081-lv-3 created 2022-02-21T18:28:48.243 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi081-lv-4 created 2022-02-21T18:28:48.251 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi140-lv-1 created 2022-02-21T18:28:48.293 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi140-lv-2 created 2022-02-21T18:28:48.318 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi140-lv-3 created 2022-02-21T18:28:48.326 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi140-lv-4 created 2022-02-21T18:28:48.334 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi179-lv-1 created 2022-02-21T18:28:48.360 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi179-lv-2 created 2022-02-21T18:28:48.368 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi179-lv-3 created 2022-02-21T18:28:48.393 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi179-lv-4 created 2022-02-21T18:28:48.468 INFO:teuthology.orchestra.run.smithi081.stdout:storageclass.storage.k8s.io/scratch created 2022-02-21T18:28:48.474 DEBUG:teuthology.orchestra.run.smithi081:> kubectl taint node smithi081 node-role.kubernetes.io/master- || true 2022-02-21T18:28:48.549 INFO:teuthology.orchestra.run.smithi081.stdout:node/smithi081 untainted 2022-02-21T18:28:48.551 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-02-21T18:28:48.551 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-02-21T18:28:48.563 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-02-21T18:28:48.564 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:48.564 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:48.572 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:48.572 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T18:28:48.579 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T18:28:48.580 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:48.580 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-02-21T18:28:48.580 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:48.580 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-02-21 18:28:46.824080087 +0000 2022-02-21T18:28:48.580 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-02-21 18:28:46.824080087 +0000 2022-02-21T18:28:48.581 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-02-21 18:28:46.824080087 +0000 2022-02-21T18:28:48.581 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2022-02-21T18:28:48.586 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-02-21T18:28:48.587 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-02-21T18:28:48.587 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000539036 s, 950 kB/s 2022-02-21T18:28:48.591 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T18:28:48.597 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T18:28:48.597 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:48.597 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-02-21T18:28:48.598 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:48.598 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-02-21 18:28:46.852080087 +0000 2022-02-21T18:28:48.598 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-02-21 18:28:46.852080087 +0000 2022-02-21T18:28:48.598 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-02-21 18:28:46.852080087 +0000 2022-02-21T18:28:48.599 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2022-02-21T18:28:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-02-21T18:28:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-02-21T18:28:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000424685 s, 1.2 MB/s 2022-02-21T18:28:48.608 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T18:28:48.656 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T18:28:48.656 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:48.656 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-02-21T18:28:48.657 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:48.657 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-02-21 18:28:46.928080085 +0000 2022-02-21T18:28:48.657 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-02-21 18:28:46.924080085 +0000 2022-02-21T18:28:48.657 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-02-21 18:28:46.924080085 +0000 2022-02-21T18:28:48.657 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2022-02-21T18:28:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-02-21T18:28:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-02-21T18:28:48.663 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00054478 s, 940 kB/s 2022-02-21T18:28:48.667 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T18:28:48.673 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T18:28:48.674 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:48.674 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-02-21T18:28:48.674 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:48.674 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-02-21 18:28:46.952080085 +0000 2022-02-21T18:28:48.674 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-02-21 18:28:46.948080085 +0000 2022-02-21T18:28:48.675 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-02-21 18:28:46.948080085 +0000 2022-02-21T18:28:48.675 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2022-02-21T18:28:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-02-21T18:28:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-02-21T18:28:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00055671 s, 920 kB/s 2022-02-21T18:28:48.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T18:28:48.799 INFO:teuthology.orchestra.run.smithi081.stdout:loop 2022-02-21T18:28:48.801 INFO:tasks.nvme_loop:Connecting nvme_loop smithi081:/dev/vg_nvme/lv_1... 2022-02-21T18:28:48.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T18:28:48.818 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-02-21T18:28:48.841 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/vg_nvme/lv_11 2022-02-21T18:28:48.862 INFO:tasks.nvme_loop:Connecting nvme_loop smithi081:/dev/vg_nvme/lv_2... 2022-02-21T18:28:48.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T18:28:48.886 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-02-21T18:28:48.914 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/vg_nvme/lv_21 2022-02-21T18:28:48.938 INFO:tasks.nvme_loop:Connecting nvme_loop smithi081:/dev/vg_nvme/lv_3... 2022-02-21T18:28:48.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T18:28:49.006 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-02-21T18:28:49.031 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/vg_nvme/lv_31 2022-02-21T18:28:49.050 INFO:tasks.nvme_loop:Connecting nvme_loop smithi081:/dev/vg_nvme/lv_4... 2022-02-21T18:28:49.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T18:28:49.070 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-02-21T18:28:49.094 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/vg_nvme/lv_41 2022-02-21T18:28:49.117 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:49.117 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:49.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo nvme list 2022-02-21T18:28:49.137 INFO:teuthology.orchestra.run.smithi081.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T18:28:49.138 INFO:teuthology.orchestra.run.smithi081.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T18:28:49.138 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme0n1 PHFT620400N7400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T18:28:49.138 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme1n1 351d13b5e00895e5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:49.139 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme2n1 2888b188cdda058c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:49.139 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme3n1 662e6b0be8d07aff Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:49.140 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-02-21T18:28:50.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo nvme list 2022-02-21T18:28:50.157 INFO:teuthology.orchestra.run.smithi081.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T18:28:50.158 INFO:teuthology.orchestra.run.smithi081.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T18:28:50.158 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme0n1 PHFT620400N7400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T18:28:50.158 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme1n1 351d13b5e00895e5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.158 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme2n1 2888b188cdda058c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.159 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme3n1 662e6b0be8d07aff Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.159 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/nvme4n1 c2f17267faedc0a3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.160 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T18:28:50.160 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:28:50.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/scratch_devs 2022-02-21T18:28:50.174 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:50.175 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:50.183 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:50.183 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T18:28:50.189 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T18:28:50.190 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.190 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-02-21T18:28:50.190 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.191 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2022-02-21 18:28:46.979775357 +0000 2022-02-21T18:28:50.191 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2022-02-21 18:28:46.979775357 +0000 2022-02-21T18:28:50.191 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2022-02-21 18:28:46.979775357 +0000 2022-02-21T18:28:50.191 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2022-02-21T18:28:50.199 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2022-02-21T18:28:50.199 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2022-02-21T18:28:50.199 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000577961 s, 886 kB/s 2022-02-21T18:28:50.203 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T18:28:50.210 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T18:28:50.210 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.210 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-02-21T18:28:50.211 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.211 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2022-02-21 18:28:47.007775358 +0000 2022-02-21T18:28:50.211 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2022-02-21 18:28:47.007775358 +0000 2022-02-21T18:28:50.211 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2022-02-21 18:28:47.007775358 +0000 2022-02-21T18:28:50.212 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2022-02-21T18:28:50.219 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2022-02-21T18:28:50.220 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2022-02-21T18:28:50.220 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000522365 s, 980 kB/s 2022-02-21T18:28:50.224 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T18:28:50.272 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T18:28:50.273 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.273 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-02-21T18:28:50.273 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.274 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2022-02-21 18:28:47.083775359 +0000 2022-02-21T18:28:50.274 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2022-02-21 18:28:47.083775359 +0000 2022-02-21T18:28:50.274 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2022-02-21 18:28:47.083775359 +0000 2022-02-21T18:28:50.274 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2022-02-21T18:28:50.283 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2022-02-21T18:28:50.284 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2022-02-21T18:28:50.284 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000493076 s, 1.0 MB/s 2022-02-21T18:28:50.288 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T18:28:50.295 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T18:28:50.295 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.295 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-02-21T18:28:50.295 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.296 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2022-02-21 18:28:47.107775360 +0000 2022-02-21T18:28:50.296 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2022-02-21 18:28:47.107775360 +0000 2022-02-21T18:28:50.296 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2022-02-21 18:28:47.107775360 +0000 2022-02-21T18:28:50.296 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2022-02-21T18:28:50.302 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2022-02-21T18:28:50.303 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2022-02-21T18:28:50.303 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000579208 s, 884 kB/s 2022-02-21T18:28:50.306 DEBUG:teuthology.orchestra.run.smithi140:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T18:28:50.388 INFO:teuthology.orchestra.run.smithi140.stdout:loop 2022-02-21T18:28:50.390 INFO:tasks.nvme_loop:Connecting nvme_loop smithi140:/dev/vg_nvme/lv_1... 2022-02-21T18:28:50.391 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T18:28:50.411 INFO:teuthology.orchestra.run.smithi140.stdout:1 2022-02-21T18:28:50.439 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/vg_nvme/lv_11 2022-02-21T18:28:50.461 INFO:tasks.nvme_loop:Connecting nvme_loop smithi140:/dev/vg_nvme/lv_2... 2022-02-21T18:28:50.462 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T18:28:50.534 INFO:teuthology.orchestra.run.smithi140.stdout:1 2022-02-21T18:28:50.557 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/vg_nvme/lv_21 2022-02-21T18:28:50.577 INFO:tasks.nvme_loop:Connecting nvme_loop smithi140:/dev/vg_nvme/lv_3... 2022-02-21T18:28:50.577 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T18:28:50.600 INFO:teuthology.orchestra.run.smithi140.stdout:1 2022-02-21T18:28:50.627 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/vg_nvme/lv_31 2022-02-21T18:28:50.649 INFO:tasks.nvme_loop:Connecting nvme_loop smithi140:/dev/vg_nvme/lv_4... 2022-02-21T18:28:50.650 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T18:28:50.669 INFO:teuthology.orchestra.run.smithi140.stdout:1 2022-02-21T18:28:50.691 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/vg_nvme/lv_41 2022-02-21T18:28:50.712 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:50.712 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:50.761 DEBUG:teuthology.orchestra.run.smithi140:> sudo nvme list 2022-02-21T18:28:50.772 INFO:teuthology.orchestra.run.smithi140.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T18:28:50.773 INFO:teuthology.orchestra.run.smithi140.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T18:28:50.773 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/nvme0n1 CVFT62330038400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T18:28:50.773 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/nvme1n1 526800795b9d9e29 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.773 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/nvme2n1 2dadabf37bf263ec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.774 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/nvme3n1 b00a9fd7feabb5a6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.774 INFO:teuthology.orchestra.run.smithi140.stdout:/dev/nvme4n1 ed82237a2921bbab Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:50.775 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T18:28:50.775 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:28:50.776 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/scratch_devs 2022-02-21T18:28:50.789 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:50.789 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:50.796 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T18:28:50.797 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T18:28:50.802 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T18:28:50.803 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.803 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-02-21T18:28:50.803 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.803 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2022-02-21 18:28:47.148960479 +0000 2022-02-21T18:28:50.804 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2022-02-21 18:28:47.148960479 +0000 2022-02-21T18:28:50.804 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2022-02-21 18:28:47.148960479 +0000 2022-02-21T18:28:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2022-02-21T18:28:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2022-02-21T18:28:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2022-02-21T18:28:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000596325 s, 859 kB/s 2022-02-21T18:28:50.816 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T18:28:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T18:28:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.864 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-02-21T18:28:50.864 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.864 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2022-02-21 18:28:47.228960477 +0000 2022-02-21T18:28:50.865 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2022-02-21 18:28:47.228960477 +0000 2022-02-21T18:28:50.865 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2022-02-21 18:28:47.228960477 +0000 2022-02-21T18:28:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2022-02-21T18:28:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2022-02-21T18:28:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2022-02-21T18:28:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000781062 s, 656 kB/s 2022-02-21T18:28:50.875 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T18:28:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T18:28:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.881 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-02-21T18:28:50.881 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.881 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2022-02-21 18:28:47.260960477 +0000 2022-02-21T18:28:50.882 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2022-02-21 18:28:47.260960477 +0000 2022-02-21T18:28:50.882 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2022-02-21 18:28:47.260960477 +0000 2022-02-21T18:28:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2022-02-21T18:28:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2022-02-21T18:28:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2022-02-21T18:28:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.00043512 s, 1.2 MB/s 2022-02-21T18:28:50.891 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T18:28:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T18:28:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T18:28:50.896 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-02-21T18:28:50.896 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T18:28:50.896 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2022-02-21 18:28:47.296960476 +0000 2022-02-21T18:28:50.897 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2022-02-21 18:28:47.296960476 +0000 2022-02-21T18:28:50.897 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2022-02-21 18:28:47.296960476 +0000 2022-02-21T18:28:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2022-02-21T18:28:50.904 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2022-02-21T18:28:50.904 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2022-02-21T18:28:50.904 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000493432 s, 1.0 MB/s 2022-02-21T18:28:50.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T18:28:50.987 INFO:teuthology.orchestra.run.smithi179.stdout:loop 2022-02-21T18:28:50.989 INFO:tasks.nvme_loop:Connecting nvme_loop smithi179:/dev/vg_nvme/lv_1... 2022-02-21T18:28:50.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T18:28:51.009 INFO:teuthology.orchestra.run.smithi179.stdout:1 2022-02-21T18:28:51.027 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/vg_nvme/lv_11 2022-02-21T18:28:51.044 INFO:tasks.nvme_loop:Connecting nvme_loop smithi179:/dev/vg_nvme/lv_2... 2022-02-21T18:28:51.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T18:28:51.109 INFO:teuthology.orchestra.run.smithi179.stdout:1 2022-02-21T18:28:51.132 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/vg_nvme/lv_21 2022-02-21T18:28:51.153 INFO:tasks.nvme_loop:Connecting nvme_loop smithi179:/dev/vg_nvme/lv_3... 2022-02-21T18:28:51.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T18:28:51.176 INFO:teuthology.orchestra.run.smithi179.stdout:1 2022-02-21T18:28:51.200 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/vg_nvme/lv_31 2022-02-21T18:28:51.220 INFO:tasks.nvme_loop:Connecting nvme_loop smithi179:/dev/vg_nvme/lv_4... 2022-02-21T18:28:51.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T18:28:51.243 INFO:teuthology.orchestra.run.smithi179.stdout:1 2022-02-21T18:28:51.262 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/vg_nvme/lv_41 2022-02-21T18:28:51.282 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:51.282 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:28:51.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo nvme list 2022-02-21T18:28:51.344 INFO:teuthology.orchestra.run.smithi179.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T18:28:51.344 INFO:teuthology.orchestra.run.smithi179.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T18:28:51.344 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/nvme0n1 CVFT6303009J400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T18:28:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/nvme1n1 c943c3028db4f3b1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/nvme2n1 4f4a7805ff33ec65 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/nvme3n1 247df244435dd54f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/nvme4n1 537aefe93ccd56e1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T18:28:51.346 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T18:28:51.347 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:28:51.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/scratch_devs 2022-02-21T18:28:51.360 INFO:teuthology.run_tasks:Running task rook... 2022-02-21T18:28:51.429 INFO:tasks.rook:Rook start 2022-02-21T18:28:51.430 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b05010b68efc838fdc35b41106e2ebdabe395a3e', 'rook_image': 'rook/ceph:master'} 2022-02-21T18:28:51.430 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e 2022-02-21T18:28:51.430 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-02-21T18:28:51.431 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-02-21T18:28:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:Cloning into 'rook'... 2022-02-21T18:29:08.748 INFO:tasks.rook:/home/teuthworker 2022-02-21T18:29:08.749 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/sbin:/usr/sbin ip addr show 2022-02-21T18:29:08.755 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-02-21T18:29:08.755 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-02-21T18:29:08.755 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2022-02-21T18:29:08.755 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-02-21T18:29:08.756 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2022-02-21T18:29:08.756 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-02-21T18:29:08.756 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T18:29:08.756 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2022-02-21T18:29:08.757 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2022-02-21T18:29:08.757 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T18:29:08.757 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2022-02-21T18:29:08.757 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2022-02-21T18:29:08.758 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T18:29:08.758 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2022-02-21T18:29:08.758 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-02-21T18:29:08.759 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2022-02-21T18:29:08.759 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-02-21T18:29:08.759 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42846sec preferred_lft 42846sec 2022-02-21T18:29:08.759 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2022-02-21T18:29:08.760 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-02-21T18:29:08.760 INFO:teuthology.orchestra.run.smithi081.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-02-21T18:29:08.760 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 02:42:4e:18:63:7a brd ff:ff:ff:ff:ff:ff 2022-02-21T18:29:08.760 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-02-21T18:29:08.761 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-02-21T18:29:08.762 DEBUG:teuthology.orchestra.run.smithi081:> which systemctl 2022-02-21T18:29:08.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/systemctl 2022-02-21T18:29:08.767 DEBUG:teuthology.orchestra.run.smithi081:> true 2022-02-21T18:29:08.845 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-02-21T18:29:08.886 INFO:teuthology.orchestra.run.smithi081.stdout:Ubuntu 2022-02-21T18:29:08.890 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-02-21T18:29:08.891 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-02-21T18:29:08.899 DEBUG:teuthology.orchestra.remote:smithi081:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T18:29:08.950 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:29:08.950 DEBUG:teuthology.orchestra.run.smithi081:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-02-21T18:29:08.959 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-02-21T18:29:09.000 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:29:09.001 DEBUG:teuthology.orchestra.run.smithi081:> dd of=operator.yaml 2022-02-21T18:29:09.008 INFO:tasks.rook:Deploying operator 2022-02-21T18:29:09.060 DEBUG:teuthology.orchestra.remote:smithi081:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T18:29:09.080 DEBUG:teuthology.orchestra.remote:smithi081:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T18:29:09.130 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-02-21T18:29:09.476 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-02-21T18:29:09.522 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-02-21T18:29:09.666 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-02-21T18:29:09.697 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-02-21T18:29:09.878 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-02-21T18:29:09.916 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-02-21T18:29:10.050 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-02-21T18:29:10.063 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-02-21T18:29:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-02-21T18:29:10.235 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-02-21T18:29:10.324 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-02-21T18:29:10.429 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-02-21T18:29:10.434 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-02-21T18:29:10.476 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-02-21T18:29:10.527 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-02-21T18:29:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-02-21T18:29:10.627 INFO:teuthology.orchestra.run.smithi081.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-02-21T18:29:10.712 INFO:teuthology.orchestra.run.smithi081.stdout:namespace/rook-ceph created 2022-02-21T18:29:10.798 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-02-21T18:29:10.826 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-02-21T18:29:10.966 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-02-21T18:29:10.993 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T18:29:11.015 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-02-21T18:29:11.057 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-02-21T18:29:11.169 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-02-21T18:29:11.173 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-02-21T18:29:11.200 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-02-21T18:29:11.232 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-02-21T18:29:11.241 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T18:29:11.307 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T18:29:11.314 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-02-21T18:29:11.316 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-02-21T18:29:11.505 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T18:29:11.508 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-02-21T18:29:11.567 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-02-21T18:29:11.735 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-02-21T18:29:11.737 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-02-21T18:29:12.188 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T18:29:12.190 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T18:29:12.529 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-02-21T18:29:12.533 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-02-21T18:29:12.559 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-02-21T18:29:12.603 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-02-21T18:29:12.834 INFO:teuthology.orchestra.run.smithi081.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-02-21T18:29:12.859 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-02-21T18:29:12.859 INFO:teuthology.orchestra.run.smithi081.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-02-21T18:29:12.899 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-02-21T18:29:12.914 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T18:29:12.934 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-02-21T18:29:13.043 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-02-21T18:29:13.075 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-02-21T18:29:13.084 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T18:29:13.092 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-02-21T18:29:13.109 INFO:teuthology.orchestra.run.smithi081.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T18:29:13.117 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-02-21T18:29:13.185 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-02-21T18:29:13.189 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-02-21T18:29:13.200 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-02-21T18:29:13.208 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-02-21T18:29:13.217 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-02-21T18:29:13.225 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-02-21T18:29:13.276 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-02-21T18:29:13.405 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-02-21T18:29:13.409 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-02-21T18:29:13.469 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T18:29:13.898 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-02-21T18:29:13.901 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-02-21T18:29:13.918 INFO:teuthology.orchestra.run.smithi081.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T18:29:13.978 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-02-21T18:29:14.089 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-ceph-mgr created 2022-02-21T18:29:14.147 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-ceph-osd created 2022-02-21T18:29:14.171 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-ceph-purge-osd created 2022-02-21T18:29:14.228 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-ceph-system created 2022-02-21T18:29:14.243 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-02-21T18:29:14.264 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-02-21T18:29:14.280 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-02-21T18:29:14.296 INFO:teuthology.orchestra.run.smithi081.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-02-21T18:29:14.322 INFO:teuthology.orchestra.run.smithi081.stdout:configmap/rook-ceph-operator-config created 2022-02-21T18:29:14.406 INFO:teuthology.orchestra.run.smithi081.stdout:deployment.apps/rook-ceph-operator created 2022-02-21T18:29:14.410 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:29:14.569 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:29:14.569 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 Pending 0 0s 2022-02-21T18:29:24.571 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:29:24.633 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:29:24.634 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 Pending 0 10s 2022-02-21T18:29:34.636 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:29:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:29:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 ContainerCreating 0 20s 2022-02-21T18:29:44.694 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:29:44.757 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:29:44.758 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 ContainerCreating 0 30s 2022-02-21T18:29:54.760 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:29:54.822 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:29:54.822 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 ContainerCreating 0 40s 2022-02-21T18:30:04.826 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:30:04.897 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:04.898 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 0/1 ContainerCreating 0 50s 2022-02-21T18:30:14.901 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T18:30:14.973 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:14.974 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-operator-75dd789779-slgjb 1/1 Running 0 60s 2022-02-21T18:30:14.976 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-slgjb 2022-02-21T18:30:14.984 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:30:14.985 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:30:15.033 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:30:15.033 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:30:15.041 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:30:15.042 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T18:30:15.052 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-02-21T18:30:15.052 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-02-21T18:30:15.052 INFO:tasks.rook: override: [mon] debug mon = 20 2022-02-21T18:30:15.053 INFO:tasks.rook: override: [mon] debug ms = 1 2022-02-21T18:30:15.053 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-02-21T18:30:15.053 INFO:tasks.rook: override: [osd] debug ms = 1 2022-02-21T18:30:15.054 INFO:tasks.rook: override: [osd] debug osd = 20 2022-02-21T18:30:15.055 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-02-21T18:30:15.057 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f - 2022-02-21T18:30:15.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.293965 I | rookcmd: starting Rook v1.8.0-alpha.0.326.gd73093883 with arguments '/usr/local/bin/rook ceph operator' 2022-02-21T18:30:15.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.294078 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-02-21T18:30:15.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.294082 I | cephcmd: starting Rook-Ceph operator 2022-02-21T18:30:15.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.623069 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-02-21T18:30:15.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.672080 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T18:30:15.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.672098 I | operator: watching all namespaces for Ceph CRs 2022-02-21T18:30:15.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.672148 I | operator: setting up schemes 2022-02-21T18:30:15.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.676576 I | operator: setting up the controller-runtime manager 2022-02-21T18:30:15.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.982208 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T18:30:15.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.984982 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T18:30:15.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985054 I | ceph-cluster-controller: successfully started 2022-02-21T18:30:15.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985146 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T18:30:15.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985178 I | ceph-crashcollector-controller: successfully started 2022-02-21T18:30:15.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985222 I | ceph-block-pool-controller: successfully started 2022-02-21T18:30:15.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985270 I | ceph-object-store-user-controller: successfully started 2022-02-21T18:30:15.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985318 I | ceph-object-realm-controller: successfully started 2022-02-21T18:30:15.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985356 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T18:30:15.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985397 I | ceph-object-zone-controller: successfully started 2022-02-21T18:30:15.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985608 I | ceph-object-controller: successfully started 2022-02-21T18:30:15.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985675 I | ceph-file-controller: successfully started 2022-02-21T18:30:15.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985736 I | ceph-nfs-controller: successfully started 2022-02-21T18:30:15.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985783 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T18:30:15.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985830 I | ceph-client-controller: successfully started 2022-02-21T18:30:15.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985877 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T18:30:15.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985910 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T18:30:15.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985935 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T18:30:15.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985958 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T18:30:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.985984 I | ceph-bucket-topic: successfully started 2022-02-21T18:30:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.986005 I | ceph-bucket-notification: successfully started 2022-02-21T18:30:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.986030 I | ceph-bucket-notification: successfully started 2022-02-21T18:30:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.986062 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T18:30:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:09.987468 I | operator: starting the controller-runtime manager 2022-02-21T18:30:15.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.089924 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T18:30:15.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.089945 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T18:30:15.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.089952 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T18:30:15.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.093750 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T18:30:15.084 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.097671 I | ceph-csi: CSI Ceph RBD driver disabled 2022-02-21T18:30:15.084 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.097700 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-02-21T18:30:15.084 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.100132 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-02-21T18:30:15.084 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.106460 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-02-21T18:30:15.085 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.106481 I | ceph-csi: CSI CephFS driver disabled 2022-02-21T18:30:15.085 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.106490 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-02-21T18:30:15.085 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.108668 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-02-21T18:30:15.085 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:10.115286 I | ceph-csi: successfully removed CSI CephFS driver 2022-02-21T18:30:16.081 INFO:teuthology.orchestra.run.smithi081.stdout:configmap/rook-config-override created 2022-02-21T18:30:16.087 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-02-21T18:30:16.087 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:30:16.088 DEBUG:teuthology.orchestra.run.smithi081:> dd of=cluster.yaml 2022-02-21T18:30:16.134 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f cluster.yaml 2022-02-21T18:30:18.561 INFO:teuthology.orchestra.run.smithi081.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-02-21T18:30:18.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.584372 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-02-21T18:30:18.574 DEBUG:teuthology.orchestra.remote:smithi081:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T18:30:18.596 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.610837 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:30:18.606 DEBUG:teuthology.orchestra.run.smithi081:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-02-21T18:30:18.669 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.684384 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-02-21T18:30:18.683 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.699179 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T18:30:18.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.707501 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:30:18.760 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775159 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T18:30:18.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775192 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T18:30:18.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775205 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T18:30:18.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775212 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T18:30:18.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775221 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T18:30:18.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775229 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T18:30:18.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775238 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T18:30:18.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775252 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T18:30:18.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775268 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T18:30:18.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775277 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T18:30:18.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775285 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T18:30:18.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775295 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T18:30:18.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775303 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T18:30:18.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775309 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T18:30:18.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775317 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T18:30:18.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775327 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T18:30:18.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775465 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:30:18.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:30:18.775483 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:30:18.871 INFO:teuthology.orchestra.run.smithi081.stdout:deployment.apps/rook-ceph-tools created 2022-02-21T18:30:18.874 INFO:tasks.rook:Waiting for tools container to start 2022-02-21T18:30:18.875 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:18.931 INFO:teuthology.orchestra.run.smithi081.stderr:No resources found in rook-ceph namespace. 2022-02-21T18:30:23.935 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:24.000 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:24.000 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 6s 2022-02-21T18:30:29.003 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:29.061 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:29.061 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 11s 2022-02-21T18:30:34.064 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:34.123 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:34.123 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 16s 2022-02-21T18:30:39.127 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:39.191 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:39.191 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 21s 2022-02-21T18:30:44.194 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:44.266 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:44.266 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 26s 2022-02-21T18:30:49.268 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:49.325 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:49.325 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 31s 2022-02-21T18:30:54.328 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:54.390 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:54.390 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 36s 2022-02-21T18:30:59.392 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:30:59.449 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:30:59.450 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 41s 2022-02-21T18:31:04.453 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:04.516 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:04.516 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 46s 2022-02-21T18:31:09.518 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:09.581 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:09.581 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 51s 2022-02-21T18:31:14.583 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 56s 2022-02-21T18:31:19.690 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:19.754 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:19.754 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 61s 2022-02-21T18:31:24.757 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:24.822 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:24.822 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 66s 2022-02-21T18:31:25.921 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938565 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T18:31:25.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938605 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T18:31:25.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938616 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:31:25.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938623 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:31:25.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938631 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:31:25.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938638 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:31:25.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938645 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:31:25.925 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938651 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:31:25.925 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938658 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:31:25.925 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938664 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:31:25.926 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938671 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:31:25.926 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938678 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:31:25.926 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938688 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:31:25.926 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938699 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:31:25.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938711 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T18:31:25.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938733 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T18:31:25.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938745 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T18:31:25.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938756 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T18:31:25.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938773 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T18:31:25.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938786 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:31:25.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938801 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:31:25.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938813 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T18:31:25.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938823 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T18:31:25.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938837 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T18:31:25.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.938860 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T18:31:25.977 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.995348 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:31:25.977 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.995376 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:31:25.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.995408 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:31:25.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:25.998482 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T18:31:25.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.006301 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:31:25.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008529 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:31:25.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008555 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:31:25.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008561 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:31:25.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008566 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:31:25.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008570 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:31:25.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008572 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:31:25.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.008575 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T18:31:26.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.088297 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:31:26.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.088329 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:31:26.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.088338 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:31:26.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.200219 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T18:31:26.205 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.223373 I | op-mon: start running mons 2022-02-21T18:31:26.324 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.342137 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:31:26.325 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.342161 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:31:26.325 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.342166 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T18:31:26.386 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.394230 I | op-mon: creating mon secrets for a new cluster 2022-02-21T18:31:26.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.416247 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:31:26.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.416274 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:31:26.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.416282 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:31:26.499 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.517500 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T18:31:26.663 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:26.678918 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T18:31:27.098 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.116209 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T18:31:27.098 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.116246 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T18:31:27.143 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.160551 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-02-21T18:31:27.436 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.453288 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-02-21T18:31:27.441 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.459075 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T18:31:27.931 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.949551 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:31:27.932 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:27.949692 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:31:29.265 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.283843 I | op-mon: parsing mon endpoints: 2022-02-21T18:31:29.266 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.283864 W | op-mon: ignoring invalid monitor 2022-02-21T18:31:29.266 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.283889 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T18:31:29.267 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.283897 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:31:29.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.284859 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T18:31:29.271 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:31:29.284918 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:31:29.825 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:29.839 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.857570 I | op-mon: targeting the mon count 3 2022-02-21T18:31:29.846 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.864203 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-02-21T18:31:29.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.944486 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-02-21T18:31:29.929 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:29.929 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 71s 2022-02-21T18:31:29.935 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:29.954110 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-02-21T18:31:30.913 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:30.931543 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi140 2022-02-21T18:31:30.914 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:30.931571 I | op-mon: mon b assigned to node smithi140 2022-02-21T18:31:30.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:30.932429 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi179 2022-02-21T18:31:30.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:30.932463 I | op-mon: mon a assigned to node smithi179 2022-02-21T18:31:31.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.205879 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi081 2022-02-21T18:31:31.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.205913 I | op-mon: mon c assigned to node smithi081 2022-02-21T18:31:31.195 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.213563 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-02-21T18:31:31.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.274283 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-02-21T18:31:31.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.686169 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-02-21T18:31:31.893 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:31.908900 I | op-mon: creating mon a 2022-02-21T18:31:32.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.096501 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:31:32.395 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.348749 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-02-21T18:31:32.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.348890 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-02-21T18:31:32.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.349156 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789"]}] data:a=10.105.182.125:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:-1] 2022-02-21T18:31:32.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.519799 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:31:32.504 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.520087 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:31:32.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:32.924732 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-02-21T18:31:33.092 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:33.110147 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-02-21T18:31:34.356 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:34.373605 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789"]}] data:a=10.105.182.125:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:0] 2022-02-21T18:31:34.356 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:34.373644 I | op-mon: waiting for mon quorum with [a] 2022-02-21T18:31:34.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:34.560361 I | op-mon: mon a is not yet running 2022-02-21T18:31:34.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:34.560383 I | op-mon: mons running: [] 2022-02-21T18:31:34.933 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:35.204 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:35.205 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 77s 2022-02-21T18:31:40.208 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:40.289 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:40.290 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 82s 2022-02-21T18:31:45.293 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:45.376 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:45.377 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 87s 2022-02-21T18:31:50.380 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:50.453 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:50.453 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 92s 2022-02-21T18:31:54.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:31:54.691932 I | op-mon: mons running: [a] 2022-02-21T18:31:55.456 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:31:55.518 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:31:55.518 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 97s 2022-02-21T18:32:00.521 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:00.898 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:00.898 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 102s 2022-02-21T18:32:05.900 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:06.150 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:06.167912 I | op-mon: Monitors in quorum: [a] 2022-02-21T18:32:06.150 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:06.167947 I | op-mon: mons created: 1 2022-02-21T18:32:06.152 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:06.152 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 108s 2022-02-21T18:32:06.371 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:06.390706 I | op-mon: waiting for mon quorum with [a] 2022-02-21T18:32:06.789 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:06.808003 I | op-mon: mons running: [a] 2022-02-21T18:32:07.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.034763 I | op-mon: Monitors in quorum: [a] 2022-02-21T18:32:07.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.034788 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:32:07.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.312685 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:32:07.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.312725 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:32:07.608 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.627724 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:32:07.609 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.627771 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:32:07.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.921026 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:32:07.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:07.921063 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:32:08.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:08.216626 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:32:08.199 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:08.216659 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:32:08.404 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:08.423129 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T18:32:08.404 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:08.423162 I | op-mon: creating mon b 2022-02-21T18:32:10.011 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:10.030246 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:32:10.469 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:10.488797 I | op-mon: mon "b" endpoint is [v2:10.104.249.171:3300,v1:10.104.249.171:6789] 2022-02-21T18:32:11.155 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:11.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:11.498580 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.249.171:6789","10.105.182.125:6789"]}] data:a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:0] 2022-02-21T18:32:11.835 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:11.836 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 113s 2022-02-21T18:32:12.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:12.235120 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:32:12.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:12.235609 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:32:13.210 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:13.230160 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T18:32:13.216 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:13.236537 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T18:32:13.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:13.236558 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T18:32:13.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:13.246595 I | op-mon: mons running: [a b] 2022-02-21T18:32:13.460 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:13.479658 I | op-mon: Monitors in quorum: [a] 2022-02-21T18:32:14.915 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:14.934593 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-02-21T18:32:15.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:15.532464 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789","10.104.249.171:6789"]}] data:a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:1] 2022-02-21T18:32:15.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:15.532502 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T18:32:16.544 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:16.563635 I | op-mon: mon b is not yet running 2022-02-21T18:32:16.544 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:16.563673 I | op-mon: mons running: [a] 2022-02-21T18:32:16.785 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:16.804473 I | op-mon: Monitors in quorum: [a] 2022-02-21T18:32:16.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:16.804504 I | op-mon: mons created: 2 2022-02-21T18:32:16.838 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:16.919 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:16.919 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 118s 2022-02-21T18:32:16.997 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:17.016705 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T18:32:17.052 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:17.072423 I | op-mon: mon b is not yet running 2022-02-21T18:32:17.053 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:17.072450 I | op-mon: mons running: [a] 2022-02-21T18:32:21.922 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:22.008 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:22.008 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 2m4s 2022-02-21T18:32:22.101 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:22.121653 I | op-mon: mon b is not yet running 2022-02-21T18:32:22.102 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:22.121687 I | op-mon: mons running: [a] 2022-02-21T18:32:27.012 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:27.205 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:27.205 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 2m9s 2022-02-21T18:32:27.411 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:27.431604 I | op-mon: mon b is not yet running 2022-02-21T18:32:27.412 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:27.431631 I | op-mon: mons running: [a] 2022-02-21T18:32:32.207 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:32.278 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:32.279 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 2m14s 2022-02-21T18:32:32.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:32.453839 I | op-mon: mon b is not yet running 2022-02-21T18:32:32.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:32.453900 I | op-mon: mons running: [a] 2022-02-21T18:32:37.281 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:37.349 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:37.349 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 0/1 ContainerCreating 0 2m19s 2022-02-21T18:32:37.446 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:37.466914 I | op-mon: mon b is not yet running 2022-02-21T18:32:37.448 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:37.466949 I | op-mon: mons running: [a] 2022-02-21T18:32:42.351 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T18:32:42.418 INFO:teuthology.orchestra.run.smithi081.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T18:32:42.418 INFO:teuthology.orchestra.run.smithi081.stdout:rook-ceph-tools-d6d7c985c-nnkrk 1/1 Running 0 2m24s 2022-02-21T18:32:42.420 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-02-21T18:32:42.420 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:32:42.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:42.498865 I | op-mon: mon b is not yet running 2022-02-21T18:32:42.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:42.498898 I | op-mon: mons running: [a] 2022-02-21T18:32:42.934 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:32:42.935 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:32:42.935 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:32:42.935 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:32:42.936 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:32:42.936 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:32:42.936 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:32:42.936 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:32:42.937 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:32:42.937 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:32:42.937 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:32:42.937 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:32:42.937 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:32:42.939 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:32:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:32:47.495 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:47.512836 I | op-mon: mon b is not yet running 2022-02-21T18:32:47.495 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:47.512863 I | op-mon: mons running: [a] 2022-02-21T18:32:52.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:52.560922 I | op-mon: mon b is not yet running 2022-02-21T18:32:52.541 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:52.560947 I | op-mon: mons running: [a] 2022-02-21T18:32:52.943 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:32:53.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:32:53.511 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:32:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:32:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:32:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:32:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:32:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:32:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:32:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:32:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:32:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:32:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:32:53.515 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:32:53.515 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:32:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:32:57.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:57.576356 I | op-mon: mon b is not yet running 2022-02-21T18:32:57.557 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:32:57.576385 I | op-mon: mons running: [a] 2022-02-21T18:33:02.583 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:02.602057 I | op-mon: mon b is not yet running 2022-02-21T18:33:02.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:02.602092 I | op-mon: mons running: [a] 2022-02-21T18:33:03.519 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:03.977 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:03.978 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:07.595 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:07.616379 I | op-mon: mon b is not yet running 2022-02-21T18:33:07.596 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:07.616413 I | op-mon: mons running: [a] 2022-02-21T18:33:12.609 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:12.630529 I | op-mon: mon b is not yet running 2022-02-21T18:33:12.609 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:12.630565 I | op-mon: mons running: [a] 2022-02-21T18:33:13.982 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:14.461 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:14.461 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:14.462 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:14.462 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:14.462 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:14.462 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:14.463 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:14.463 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:14.463 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:14.463 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:14.464 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:14.464 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:14.464 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:14.465 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:17.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:17.644196 I | op-mon: mon b is not yet running 2022-02-21T18:33:17.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:17.644228 I | op-mon: mons running: [a] 2022-02-21T18:33:22.637 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:22.658966 I | op-mon: mon b is not yet running 2022-02-21T18:33:22.637 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:22.659000 I | op-mon: mons running: [a] 2022-02-21T18:33:24.468 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:24.951 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:24.952 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:24.952 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:24.953 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:24.953 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:24.953 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:24.955 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:24.957 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:24.957 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:24.960 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:27.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:27.675169 I | op-mon: mon b is not yet running 2022-02-21T18:33:27.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:27.675204 I | op-mon: mons running: [a] 2022-02-21T18:33:32.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:32.690072 I | op-mon: mon b is not yet running 2022-02-21T18:33:32.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:32.690111 I | op-mon: mons running: [a] 2022-02-21T18:33:34.962 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:35.555 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:35.555 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:35.556 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:35.556 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:35.556 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:35.556 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:35.557 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:35.557 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:35.557 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:35.557 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:35.558 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:35.558 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:35.558 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:35.559 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:37.682 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:37.704788 I | op-mon: mon b is not yet running 2022-02-21T18:33:37.683 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:37.704825 I | op-mon: mons running: [a] 2022-02-21T18:33:42.696 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:42.718311 I | op-mon: mon b is not yet running 2022-02-21T18:33:42.696 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:42.718343 I | op-mon: mons running: [a] 2022-02-21T18:33:45.566 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:46.033 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:46.036 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:46.036 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:46.036 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:46.036 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:46.037 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:47.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:47.734024 I | op-mon: mon b is not yet running 2022-02-21T18:33:47.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:47.734062 I | op-mon: mons running: [a] 2022-02-21T18:33:52.726 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:52.748551 I | op-mon: mon b is not yet running 2022-02-21T18:33:52.726 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:52.748587 I | op-mon: mons running: [a] 2022-02-21T18:33:56.041 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:33:56.522 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:33:56.523 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:33:56.523 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:33:56.523 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:33:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:33:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:33:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:33:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:33:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:33:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:33:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:33:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:33:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:33:56.527 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:33:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:33:57.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:57.803135 I | op-mon: mon b is not yet running 2022-02-21T18:33:57.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:33:57.803172 I | op-mon: mons running: [a] 2022-02-21T18:34:02.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:02.819786 I | op-mon: mon b is not yet running 2022-02-21T18:34:02.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:02.819821 I | op-mon: mons running: [a] 2022-02-21T18:34:06.531 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:07.027 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:07.027 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:07.030 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:07.030 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:07.030 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:34:07.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:07.834713 I | op-mon: mon b is not yet running 2022-02-21T18:34:07.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:07.834750 I | op-mon: mons running: [a] 2022-02-21T18:34:12.827 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:12.850018 I | op-mon: mon b is not yet running 2022-02-21T18:34:12.827 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:12.850053 I | op-mon: mons running: [a] 2022-02-21T18:34:17.034 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:17.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:17.518 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:17.518 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:17.518 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:17.519 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:17.519 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:17.519 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:17.519 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:17.519 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:34:17.842 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:17.866203 I | op-mon: mon b is not yet running 2022-02-21T18:34:17.843 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:17.866239 I | op-mon: mons running: [a] 2022-02-21T18:34:22.860 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:22.884223 I | op-mon: mon b is not yet running 2022-02-21T18:34:22.861 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:22.884251 I | op-mon: mons running: [a] 2022-02-21T18:34:27.524 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:27.875 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:27.899271 I | op-mon: mon b is not yet running 2022-02-21T18:34:27.875 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:27.899301 I | op-mon: mons running: [a] 2022-02-21T18:34:28.036 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:28.036 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:28.038 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:28.038 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:28.038 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:28.038 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:28.039 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:28.039 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:28.039 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:34:32.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:32.914109 I | op-mon: mon b is not yet running 2022-02-21T18:34:32.891 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:32.914145 I | op-mon: mons running: [a] 2022-02-21T18:34:37.907 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:37.929194 I | op-mon: mon b is not yet running 2022-02-21T18:34:37.907 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:37.929232 I | op-mon: mons running: [a] 2022-02-21T18:34:38.044 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:38.552 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:34:42.921 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:42.945637 I | op-mon: mon b is not yet running 2022-02-21T18:34:42.922 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:42.945671 I | op-mon: mons running: [a] 2022-02-21T18:34:42.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:42.953896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to start mon pods: failed to wait for mon quorum: exceeded max retry count waiting for monitors to reach quorum 2022-02-21T18:34:42.935 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:42.959751 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:34:42.971 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:42.995748 I | op-mon: parsing mon endpoints: a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:34:42.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:43.012262 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:34:48.118 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.142396 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:34:48.118 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.142424 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:34:48.118 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.142439 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:34:48.131 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.155916 I | op-mon: parsing mon endpoints: a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:34:48.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.158224 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:34:48.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.158479 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:34:48.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.403331 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:34:48.507 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.532316 I | op-mon: start running mons 2022-02-21T18:34:48.514 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.538722 I | op-mon: parsing mon endpoints: a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:34:48.559 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:48.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:48.671858 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789","10.104.249.171:6789"]}] data:a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:1] 2022-02-21T18:34:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:49.043 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:49.043 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:49.043 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:49.044 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:49.044 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:49.044 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:49.044 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:49.045 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:49.045 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:49.045 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:49.045 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:49.046 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:49.047 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:34:49.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:49.270467 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:34:49.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:49.270705 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:34:50.844 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:50.868527 I | op-mon: targeting the mon count 3 2022-02-21T18:34:50.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:50.872873 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:34:51.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.106016 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:34:51.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.106051 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:34:51.319 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.343518 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:34:51.319 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.343555 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:34:51.545 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.570268 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:34:51.546 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.570302 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:34:51.776 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.800931 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:34:51.776 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:51.800958 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:34:51.994 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:52.019033 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T18:34:51.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:52.019067 I | op-mon: creating mon c 2022-02-21T18:34:52.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:52.132723 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:34:52.464 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:52.488666 I | op-mon: mon "b" endpoint is [v2:10.104.249.171:3300,v1:10.104.249.171:6789] 2022-02-21T18:34:52.738 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:52.763280 I | op-mon: mon "c" endpoint is [v2:10.105.216.166:3300,v1:10.105.216.166:6789] 2022-02-21T18:34:53.244 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:53.269460 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789","10.104.249.171:6789","10.105.216.166:6789"]}] data:a=10.105.182.125:6789,b=10.104.249.171:6789,c=10.105.216.166:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:1] 2022-02-21T18:34:53.846 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:53.871205 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:34:53.850 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:53.871535 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:34:54.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.278163 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-02-21T18:34:54.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.284602 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T18:34:54.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.293367 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T18:34:54.272 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.297699 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T18:34:54.285 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.310316 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T18:34:54.463 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:54.487801 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-02-21T18:34:55.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:55.307278 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.182.125:6789","10.104.249.171:6789","10.105.216.166:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:34:55.283 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:55.307318 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T18:34:55.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:55.672412 I | op-mon: mon b is not yet running 2022-02-21T18:34:55.847 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:55.871599 I | op-mon: mon c is not yet running 2022-02-21T18:34:55.847 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:55.871633 I | op-mon: mons running: [a] 2022-02-21T18:34:56.109 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.133875 I | op-mon: Monitors in quorum: [a] 2022-02-21T18:34:56.109 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.133903 I | op-mon: mons created: 3 2022-02-21T18:34:56.362 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.387323 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T18:34:56.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.400594 I | op-mon: mon b is not yet running 2022-02-21T18:34:56.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.478173 I | op-mon: mon c is not yet running 2022-02-21T18:34:56.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:34:56.478204 I | op-mon: mons running: [a] 2022-02-21T18:34:59.050 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:34:59.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:34:59.589 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:34:59.589 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:34:59.589 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:34:59.592 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:34:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:01.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:01.492828 I | op-mon: mon b is not yet running 2022-02-21T18:35:01.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:01.498236 I | op-mon: mon c is not yet running 2022-02-21T18:35:01.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:01.498255 I | op-mon: mons running: [a] 2022-02-21T18:35:06.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:06.514967 I | op-mon: mon b is not yet running 2022-02-21T18:35:06.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:06.520159 I | op-mon: mon c is not yet running 2022-02-21T18:35:06.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:06.520185 I | op-mon: mons running: [a] 2022-02-21T18:35:09.596 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:35:10.096 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:35:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:35:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:35:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:35:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:35:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:35:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:35:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:35:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:35:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:35:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:35:10.100 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:35:10.100 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:35:10.101 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:35:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:11.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:11.538569 I | op-mon: mon b is not yet running 2022-02-21T18:35:11.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:11.545797 I | op-mon: mon c is not yet running 2022-02-21T18:35:11.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:11.545844 I | op-mon: mons running: [a] 2022-02-21T18:35:16.539 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:16.564398 I | op-mon: mon b is not yet running 2022-02-21T18:35:16.544 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:16.570461 I | op-mon: mon c is not yet running 2022-02-21T18:35:16.545 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:16.570489 I | op-mon: mons running: [a] 2022-02-21T18:35:20.105 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:35:21.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:21.595353 I | op-mon: mon c is not yet running 2022-02-21T18:35:21.571 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:21.595382 I | op-mon: mons running: [a b] 2022-02-21T18:35:24.904 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:35:24.905 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:35:24.905 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:35:24.905 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:35:24.905 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:35:24.906 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:35:24.906 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:35:24.906 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:35:24.906 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:35:24.907 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:35:24.907 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:35:24.907 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:35:24.907 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:35:24.908 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:35:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:26.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:26.618040 I | op-mon: mon c is not yet running 2022-02-21T18:35:26.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:26.618070 I | op-mon: mons running: [a b] 2022-02-21T18:35:31.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:31.799354 I | op-mon: mon c is not yet running 2022-02-21T18:35:31.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:31.799385 I | op-mon: mons running: [a b] 2022-02-21T18:35:34.913 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:35:35.449 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:35:35.450 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:35:35.450 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:35:35.450 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:35:35.450 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:35:35.451 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:35:35.451 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:35:35.451 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:35:35.452 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:35:35.452 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:35:35.452 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:35:35.452 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:35:35.452 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:35:35.455 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:35:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:36.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:36.819983 I | op-mon: mon c is not yet running 2022-02-21T18:35:36.800 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:36.820030 I | op-mon: mons running: [a b] 2022-02-21T18:35:41.822 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:41.845974 I | op-mon: mon c is not yet running 2022-02-21T18:35:41.822 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:41.846004 I | op-mon: mons running: [a b] 2022-02-21T18:35:45.459 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:35:45.930 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:35:45.931 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:35:45.931 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:35:45.931 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:35:45.931 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:35:45.932 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:35:45.932 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:35:45.932 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:35:45.932 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:35:45.933 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:35:45.933 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:35:45.933 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:35:45.933 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:35:45.935 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:35:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:46.843 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:46.867136 I | op-mon: mon c is not yet running 2022-02-21T18:35:46.843 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:46.867167 I | op-mon: mons running: [a b] 2022-02-21T18:35:51.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:51.893385 I | op-mon: mon c is not yet running 2022-02-21T18:35:51.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:51.893416 I | op-mon: mons running: [a b] 2022-02-21T18:35:55.939 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:35:56.415 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:35:56.416 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:35:56.416 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:35:56.416 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:35:56.417 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:35:56.417 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:35:56.417 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:35:56.417 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:35:56.418 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:35:56.418 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:35:56.418 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:35:56.418 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:35:56.419 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:35:56.419 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:35:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:35:56.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:56.913386 I | op-mon: mon c is not yet running 2022-02-21T18:35:56.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:35:56.913422 I | op-mon: mons running: [a b] 2022-02-21T18:36:01.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:01.938436 I | op-mon: mon c is not yet running 2022-02-21T18:36:01.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:01.938466 I | op-mon: mons running: [a b] 2022-02-21T18:36:06.424 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:36:06.937 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:06.960355 I | op-mon: mon c is not yet running 2022-02-21T18:36:06.938 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:06.960391 I | op-mon: mons running: [a b] 2022-02-21T18:36:07.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:36:07.181 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:36:07.182 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:36:07.182 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:36:07.183 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:36:07.183 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:36:07.183 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:36:07.184 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:36:07.184 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:36:07.184 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:36:07.184 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:36:07.185 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:36:07.185 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:36:07.185 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:36:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:36:11.960 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:11.981875 I | op-mon: mon c is not yet running 2022-02-21T18:36:11.960 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:11.981915 I | op-mon: mons running: [a b] 2022-02-21T18:36:16.986 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:17.008856 I | op-mon: mon c is not yet running 2022-02-21T18:36:16.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:17.008894 I | op-mon: mons running: [a b] 2022-02-21T18:36:17.191 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:36:17.720 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:36:17.721 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:36:17.721 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:36:17.722 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:36:17.722 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:36:17.723 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:36:17.723 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:36:17.723 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:36:17.724 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:36:17.724 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:36:17.725 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:36:17.725 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:36:17.726 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:36:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:36:17.728 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:36:22.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:22.029175 I | op-mon: mon c is not yet running 2022-02-21T18:36:22.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:22.029210 I | op-mon: mons running: [a b] 2022-02-21T18:36:27.034 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:27.054338 I | op-mon: mon c is not yet running 2022-02-21T18:36:27.035 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:27.054367 I | op-mon: mons running: [a b] 2022-02-21T18:36:27.730 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:36:28.196 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:36:28.197 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:36:28.197 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:36:28.197 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:36:28.197 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:36:28.198 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:36:28.198 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:36:28.198 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:36:28.198 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:36:28.199 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:36:28.199 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:36:28.199 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:36:28.200 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:36:28.201 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:36:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:36:33.485 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:33.507287 I | op-mon: mon c is not yet running 2022-02-21T18:36:33.485 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:33.507329 I | op-mon: mons running: [a b] 2022-02-21T18:36:38.204 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:36:38.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:39.002259 I | op-mon: mon c is not yet running 2022-02-21T18:36:38.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:39.002292 I | op-mon: mons running: [a b] 2022-02-21T18:36:39.387 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:36:39.388 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:36:39.388 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:36:39.388 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:36:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:36:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:36:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:36:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:36:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:36:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:36:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:36:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:36:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:36:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:36:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:36:44.027 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:44.048995 I | op-mon: mon c is not yet running 2022-02-21T18:36:44.027 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:44.049034 I | op-mon: mons running: [a b] 2022-02-21T18:36:49.056 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:49.071638 I | op-mon: mons running: [a b c] 2022-02-21T18:36:49.397 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:36:53.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:36:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:36:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:36:53.442 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:36:53.442 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:36:53.442 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:36:53.442 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:36:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:36:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:36:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:36:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:36:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:36:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:36:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:36:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:36:53.693 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:53.714736 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:36:53.694 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:53.714781 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T18:36:54.017 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:54.038345 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T18:36:54.367 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:54.387867 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T18:36:54.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:54.783014 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T18:36:55.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.248105 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T18:36:55.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.248129 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T18:36:55.654 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.675473 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T18:36:55.882 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.902281 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T18:36:55.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.902319 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T18:36:55.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:55.902327 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T18:36:56.244 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:56.265008 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T18:36:56.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:56.283028 I | op-mgr: start running mgr 2022-02-21T18:36:56.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:56.283067 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T18:36:57.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:57.091703 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi081": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:36:57.119 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:36:57.140150 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi081": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:37:03.359 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:03.379160 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi179": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:37:03.387 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:03.408689 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi179": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:37:03.451 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:37:03.917 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:03.918 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2022-02-21T18:37:03.918 INFO:teuthology.orchestra.run.smithi081.stderr:pg stat 2022-02-21T18:37:03.918 INFO:teuthology.orchestra.run.smithi081.stderr:pg getmap 2022-02-21T18:37:03.918 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T18:37:03.919 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T18:37:03.919 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_pools_json 2022-02-21T18:37:03.919 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-pool [...] 2022-02-21T18:37:03.919 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-primary [] [...] 2022-02-21T18:37:03.919 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls-by-osd [] [...] 2022-02-21T18:37:03.920 INFO:teuthology.orchestra.run.smithi081.stderr:pg ls [] [...] 2022-02-21T18:37:03.920 INFO:teuthology.orchestra.run.smithi081.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T18:37:03.920 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2022-02-21T18:37:03.923 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 22 2022-02-21T18:37:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T18:37:13.927 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:37:14.477 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:14.480 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-02-21T18:37:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 2 2022-02-21T18:37:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-02-21T18:37:17.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:17.990544 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-02-21T18:37:17.974 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:17.995486 I | op-mgr: setting services to point to mgr "a" 2022-02-21T18:37:18.014 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.034795 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T18:37:18.014 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.034927 I | op-mgr: successful modules: balancer 2022-02-21T18:37:18.041 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.062167 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T18:37:18.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.062196 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-02-21T18:37:18.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.062206 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T18:37:18.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.098444 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T18:37:18.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.102300 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-02-21T18:37:18.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.102329 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T18:37:18.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.102334 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T18:37:18.493 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.513775 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T18:37:18.493 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.513811 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:37:18.512 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.533208 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-02-21T18:37:18.512 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.533238 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-02-21T18:37:18.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:18.533251 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-02-21T18:37:19.112 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:19.132943 I | op-mgr: successful modules: dashboard 2022-02-21T18:37:20.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:20.249093 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T18:37:20.233 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:20.252632 I | op-mgr: successful modules: prometheus 2022-02-21T18:37:24.494 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:37:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:28.644 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-02-21T18:37:28.648 INFO:teuthology.orchestra.run.smithi081.stderr:command terminated with exit code 2 2022-02-21T18:37:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-02-21T18:37:28.886 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:28.905835 I | op-mgr: successful modules: orchestrator modules 2022-02-21T18:37:38.652 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status -f json 2022-02-21T18:37:39.272 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:39.272 INFO:teuthology.orchestra.run.smithi081.stdout:{"available": true, "backend": "rook"} 2022-02-21T18:37:39.290 INFO:tasks.rook: mgr/rook orchestrator is active 2022-02-21T18:37:39.291 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph config set mgr mgr/rook/storage_class scratch 2022-02-21T18:37:39.957 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch apply osd --all-available-devices 2022-02-21T18:37:40.784 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.803855 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T18:37:40.784 INFO:tasks.rook.operator.smithi081.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T18:37:40.785 INFO:tasks.rook.operator.smithi081.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T18:37:40.785 INFO:tasks.rook.operator.smithi081.stdout: ... // 3 identical fields 2022-02-21T18:37:40.785 INFO:tasks.rook.operator.smithi081.stdout: Config: nil, 2022-02-21T18:37:40.786 INFO:teuthology.orchestra.run.smithi081.stdout:['Success'] 2022-02-21T18:37:40.787 INFO:tasks.rook.operator.smithi081.stdout: Selection: {}, 2022-02-21T18:37:40.787 INFO:tasks.rook.operator.smithi081.stdout:- StorageClassDeviceSets: nil, 2022-02-21T18:37:40.787 INFO:tasks.rook.operator.smithi081.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T18:37:40.788 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.788 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi081-lv-1", 2022-02-21T18:37:40.788 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.789 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.795 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.796 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.796 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi081-lv-2", 2022-02-21T18:37:40.796 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.796 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.797 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.797 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.797 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi081-lv-3", 2022-02-21T18:37:40.797 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.798 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.798 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.798 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.798 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi081-lv-4", 2022-02-21T18:37:40.798 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.799 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.799 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.799 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.799 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi140-lv-1", 2022-02-21T18:37:40.800 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.800 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.800 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.800 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.800 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi140-lv-2", 2022-02-21T18:37:40.801 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.801 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.801 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.801 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.802 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi140-lv-3", 2022-02-21T18:37:40.802 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.802 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.802 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.802 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.803 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi140-lv-4", 2022-02-21T18:37:40.803 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.803 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.803 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.804 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.804 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi179-lv-1", 2022-02-21T18:37:40.804 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.804 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.804 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.805 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.805 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi179-lv-2", 2022-02-21T18:37:40.805 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.805 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.805 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.806 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.806 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi179-lv-3", 2022-02-21T18:37:40.806 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.806 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.807 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.807 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:37:40.807 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi179-lv-4", 2022-02-21T18:37:40.807 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:37:40.807 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T18:37:40.808 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.808 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:37:40.808 INFO:tasks.rook.operator.smithi081.stdout: }, 2022-02-21T18:37:40.808 INFO:tasks.rook.operator.smithi081.stdout: Annotations: nil, 2022-02-21T18:37:40.809 INFO:tasks.rook.operator.smithi081.stdout: Labels: nil, 2022-02-21T18:37:40.809 INFO:tasks.rook.operator.smithi081.stdout: ... // 20 identical fields 2022-02-21T18:37:40.809 INFO:tasks.rook.operator.smithi081.stdout: } 2022-02-21T18:37:40.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.803969 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T18:37:40.810 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:37:40.804458 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T18:37:40.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.804577 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-02-21T18:37:40.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.804601 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-02-21T18:37:40.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.804613 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-02-21T18:37:40.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.804673 I | operator: successfully started the controller-runtime manager 2022-02-21T18:37:40.812 INFO:tasks.rook:Waiting for 12 OSDs 2022-02-21T18:37:40.812 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:37:40.840 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.860940 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T18:37:40.841 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.860974 I | operator: watching all namespaces for Ceph CRs 2022-02-21T18:37:40.841 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.861023 I | operator: setting up schemes 2022-02-21T18:37:40.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:40.865431 I | operator: setting up the controller-runtime manager 2022-02-21T18:37:41.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.168183 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T18:37:41.152 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.170818 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T18:37:41.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.170861 I | ceph-cluster-controller: successfully started 2022-02-21T18:37:41.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.170916 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T18:37:41.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.170945 I | ceph-crashcollector-controller: successfully started 2022-02-21T18:37:41.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.170983 I | ceph-block-pool-controller: successfully started 2022-02-21T18:37:41.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171027 I | ceph-object-store-user-controller: successfully started 2022-02-21T18:37:41.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171064 I | ceph-object-realm-controller: successfully started 2022-02-21T18:37:41.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171094 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T18:37:41.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171122 I | ceph-object-zone-controller: successfully started 2022-02-21T18:37:41.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171302 I | ceph-object-controller: successfully started 2022-02-21T18:37:41.157 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171362 I | ceph-file-controller: successfully started 2022-02-21T18:37:41.157 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171441 I | ceph-nfs-controller: successfully started 2022-02-21T18:37:41.158 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171492 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T18:37:41.158 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171535 I | ceph-client-controller: successfully started 2022-02-21T18:37:41.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171574 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T18:37:41.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171619 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T18:37:41.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171652 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T18:37:41.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171685 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T18:37:41.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171703 I | ceph-bucket-topic: successfully started 2022-02-21T18:37:41.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171718 I | ceph-bucket-notification: successfully started 2022-02-21T18:37:41.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171733 I | ceph-bucket-notification: successfully started 2022-02-21T18:37:41.161 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.171751 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T18:37:41.161 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.172696 I | operator: starting the controller-runtime manager 2022-02-21T18:37:41.267 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.285791 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T18:37:41.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.285871 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T18:37:41.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.285891 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T18:37:41.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.286111 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:37:41.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.291053 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T18:37:41.272 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.292680 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:37:41.272 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.292722 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T18:37:41.273 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.292730 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:37:41.273 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.292812 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:37:41.274 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.293599 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T18:37:41.274 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:37:41.293643 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:37:41.294 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.315453 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:37:41.301 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:41.321 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T18:37:41.321 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:37:41.391 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.410922 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T18:37:41.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414650 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T18:37:41.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414684 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T18:37:41.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414702 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T18:37:41.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414714 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T18:37:41.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414726 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T18:37:41.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414738 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T18:37:41.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414752 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T18:37:41.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414766 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T18:37:41.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414778 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T18:37:41.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414793 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T18:37:41.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414804 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T18:37:41.404 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414818 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T18:37:41.405 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414830 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T18:37:41.405 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414840 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T18:37:41.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414850 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T18:37:41.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414862 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T18:37:41.407 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414957 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:37:41.407 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:41.414976 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:37:47.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597241 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T18:37:47.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597296 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T18:37:47.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597312 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:37:47.578 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597323 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:37:47.578 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597339 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:37:47.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597351 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:37:47.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597364 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:37:47.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597376 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:37:47.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597390 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:37:47.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597402 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:37:47.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597416 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:37:47.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597428 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:37:47.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597440 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:37:47.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597452 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:37:47.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597465 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T18:37:47.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597477 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T18:37:47.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597488 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T18:37:47.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597500 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T18:37:47.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597513 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T18:37:47.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597526 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:37:47.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597540 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:37:47.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597552 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T18:37:47.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597564 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T18:37:47.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597590 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T18:37:47.585 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.597602 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T18:37:47.585 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.602395 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T18:37:47.588 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609170 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:37:47.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609190 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:37:47.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609195 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:37:47.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609199 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:37:47.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609203 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:37:47.590 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609207 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:37:47.590 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.609211 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T18:37:47.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.819513 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:37:47.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.819542 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:37:47.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.819549 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:37:47.961 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:47.981080 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T18:37:48.167 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.187655 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:37:48.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.187704 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:37:48.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.187717 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T18:37:48.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.850042 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:37:48.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.850074 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:37:48.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.850091 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:37:48.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.929812 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:37:48.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.929842 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:37:48.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:48.929854 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:37:48.999 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.019745 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:37:49.143 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.158754 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:37:49.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.159086 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:37:49.151 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.170367 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T18:37:49.380 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.400970 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T18:37:49.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.484447 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T18:37:49.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.484472 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T18:37:49.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.499269 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T18:37:49.543 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.564117 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:37:49.633 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.653930 I | op-mon: start running mons 2022-02-21T18:37:49.688 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.709320 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:37:49.770 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:49.785838 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:a=10.105.182.125:6789,b=10.104.249.171:6789,c=10.105.216.166:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:37:50.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:50.409472 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:37:50.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:50.409763 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:37:51.326 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:37:51.921 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:37:51.941 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T18:37:51.942 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:37:51.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:51.990750 I | op-mon: targeting the mon count 3 2022-02-21T18:37:51.974 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:51.995173 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:37:52.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:52.335281 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:37:52.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:52.335318 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:37:52.657 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:52.677730 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:37:52.657 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:52.677765 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:37:53.020 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.041394 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:37:53.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.041426 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:37:53.354 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.374152 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:37:53.354 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.374193 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:37:53.783 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.803494 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T18:37:53.783 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.803534 I | op-mon: checking for basic quorum with existing mons 2022-02-21T18:37:53.921 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:53.942224 I | op-mon: mon "c" endpoint is [v2:10.105.216.166:3300,v1:10.105.216.166:6789] 2022-02-21T18:37:54.152 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:54.172999 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:37:54.447 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:54.467839 I | op-mon: mon "b" endpoint is [v2:10.104.249.171:3300,v1:10.104.249.171:6789] 2022-02-21T18:37:54.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:54.986842 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:37:55.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:55.585004 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:37:55.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:55.585368 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:37:55.984 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:56.004812 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T18:37:55.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:56.016338 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T18:37:55.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:56.016369 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:37:56.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:56.591104 I | op-mon: mons running: [c a b] 2022-02-21T18:37:57.008 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.027292 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:37:57.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.035871 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T18:37:57.025 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.046408 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T18:37:57.026 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.046440 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:37:57.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.199801 I | op-mon: mons running: [c a b] 2022-02-21T18:37:57.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.646319 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:37:57.629 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.651044 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T18:37:57.636 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.657383 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T18:37:57.636 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.657399 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:37:57.768 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:57.789259 I | op-mon: mons running: [c a b] 2022-02-21T18:37:58.203 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:58.224210 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:37:58.204 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:58.224240 I | op-mon: mons created: 3 2022-02-21T18:37:58.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:58.624317 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:37:58.629 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:58.650050 I | op-mon: mons running: [c a b] 2022-02-21T18:37:59.051 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:59.071762 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:37:59.052 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:59.071811 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T18:37:59.452 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:59.472652 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T18:37:59.933 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:37:59.953596 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T18:38:00.418 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:00.438834 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T18:38:00.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:00.882978 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T18:38:00.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:00.883021 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T18:38:01.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:01.302768 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T18:38:01.703 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:01.724281 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T18:38:01.704 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:01.724340 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T18:38:01.704 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:01.724351 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T18:38:01.943 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:02.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.155824 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T18:38:02.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.215202 I | op-mgr: start running mgr 2022-02-21T18:38:02.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.215258 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T18:38:02.443 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:02.462 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T18:38:02.462 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:38:02.706 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.727312 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T18:38:02.716 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.737888 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T18:38:02.720 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.741519 I | op-mgr: setting services to point to mgr "a" 2022-02-21T18:38:02.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.866277 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T18:38:02.846 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.866399 I | op-mgr: successful modules: balancer 2022-02-21T18:38:02.859 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.880515 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T18:38:02.860 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.880556 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T18:38:02.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:02.950650 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T18:38:03.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.066614 I | op-osd: creating 1 new PVCs for device set "smithi081-lv-1" 2022-02-21T18:38:03.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.270297 I | op-osd: successfully provisioned PVC "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:38:03.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.270336 I | op-osd: creating 1 new PVCs for device set "smithi081-lv-2" 2022-02-21T18:38:03.447 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.467949 I | op-osd: successfully provisioned PVC "smithi081-lv-2-data-08pkzc" 2022-02-21T18:38:03.447 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.467984 I | op-osd: creating 1 new PVCs for device set "smithi081-lv-3" 2022-02-21T18:38:03.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.688553 I | op-mgr: successful modules: dashboard 2022-02-21T18:38:03.674 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.695494 I | op-mgr: successful modules: prometheus 2022-02-21T18:38:03.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.696100 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T18:38:03.720 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.741094 I | op-osd: successfully provisioned PVC "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:38:03.720 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.741132 I | op-osd: creating 1 new PVCs for device set "smithi081-lv-4" 2022-02-21T18:38:03.904 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.925291 I | op-osd: successfully provisioned PVC "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:38:03.905 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:03.925329 I | op-osd: creating 1 new PVCs for device set "smithi140-lv-1" 2022-02-21T18:38:04.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.167347 I | op-osd: successfully provisioned PVC "smithi140-lv-1-data-0sfl8x" 2022-02-21T18:38:04.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.167389 I | op-osd: creating 1 new PVCs for device set "smithi140-lv-2" 2022-02-21T18:38:04.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.274288 I | op-osd: successfully provisioned PVC "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:38:04.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.274320 I | op-osd: creating 1 new PVCs for device set "smithi140-lv-3" 2022-02-21T18:38:04.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.470473 I | op-osd: successfully provisioned PVC "smithi140-lv-3-data-07nf5l" 2022-02-21T18:38:04.450 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.470510 I | op-osd: creating 1 new PVCs for device set "smithi140-lv-4" 2022-02-21T18:38:04.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.712287 I | op-osd: successfully provisioned PVC "smithi140-lv-4-data-05zd97" 2022-02-21T18:38:04.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.712321 I | op-osd: creating 1 new PVCs for device set "smithi179-lv-1" 2022-02-21T18:38:04.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.891202 I | op-osd: successfully provisioned PVC "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:38:04.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:04.891248 I | op-osd: creating 1 new PVCs for device set "smithi179-lv-2" 2022-02-21T18:38:05.048 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.069330 I | op-osd: successfully provisioned PVC "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:38:05.049 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.069365 I | op-osd: creating 1 new PVCs for device set "smithi179-lv-3" 2022-02-21T18:38:05.247 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.268708 I | op-osd: successfully provisioned PVC "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:38:05.248 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.268737 I | op-osd: creating 1 new PVCs for device set "smithi179-lv-4" 2022-02-21T18:38:05.362 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.383324 I | op-mgr: successful modules: orchestrator modules 2022-02-21T18:38:05.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.470084 I | op-osd: successfully provisioned PVC "smithi179-lv-4-data-0thzk9" 2022-02-21T18:38:05.459 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.480479 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:38:05.887 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.907818 I | op-osd: started OSD provisioning job for PVC "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:38:05.887 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:05.907852 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:38:06.265 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:06.285835 I | op-osd: started OSD provisioning job for PVC "smithi081-lv-2-data-08pkzc" 2022-02-21T18:38:06.267 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:06.285870 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:38:06.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:06.701009 I | op-osd: started OSD provisioning job for PVC "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:38:06.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:06.701042 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:38:07.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:07.180300 I | op-osd: started OSD provisioning job for PVC "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:38:07.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:07.180335 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0sfl8x" 2022-02-21T18:38:07.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:07.590921 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-1-data-0sfl8x" 2022-02-21T18:38:07.571 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:07.590957 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:38:08.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:08.731905 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:38:08.712 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:08.731957 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:38:09.506 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:09.527244 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-3-data-07nf5l" 2022-02-21T18:38:09.507 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:09.527287 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:38:09.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:09.928884 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-4-data-05zd97" 2022-02-21T18:38:09.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:09.928928 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:38:10.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.203353 I | op-osd: started OSD provisioning job for PVC "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:38:10.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.203383 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:38:10.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.411447 I | op-osd: started OSD provisioning job for PVC "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:38:10.391 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.411469 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:38:10.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.694154 I | op-osd: started OSD provisioning job for PVC "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:38:10.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.694199 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:38:10.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.846176 I | op-osd: started OSD provisioning job for PVC "smithi179-lv-4-data-0thzk9" 2022-02-21T18:38:10.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.846201 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T18:38:10.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:10.846205 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T18:38:11.041 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061015 I | op-osd: OSD orchestration status for PVC smithi081-lv-1-data-0zzmvw is "starting" 2022-02-21T18:38:11.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061045 I | op-osd: OSD orchestration status for PVC smithi081-lv-2-data-08pkzc is "starting" 2022-02-21T18:38:11.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061055 I | op-osd: OSD orchestration status for PVC smithi081-lv-3-data-0tvdqv is "starting" 2022-02-21T18:38:11.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061062 I | op-osd: OSD orchestration status for PVC smithi081-lv-4-data-0tqs4h is "starting" 2022-02-21T18:38:11.043 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061070 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0sfl8x is "starting" 2022-02-21T18:38:11.043 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061080 I | op-osd: OSD orchestration status for PVC smithi140-lv-2-data-0s5fp2 is "starting" 2022-02-21T18:38:11.044 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061089 I | op-osd: OSD orchestration status for PVC smithi140-lv-3-data-07nf5l is "starting" 2022-02-21T18:38:11.044 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061097 I | op-osd: OSD orchestration status for PVC smithi140-lv-4-data-05zd97 is "starting" 2022-02-21T18:38:11.044 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061105 I | op-osd: OSD orchestration status for PVC smithi179-lv-1-data-0gqt5r is "starting" 2022-02-21T18:38:11.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061114 I | op-osd: OSD orchestration status for PVC smithi179-lv-2-data-0bk9j2 is "starting" 2022-02-21T18:38:11.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061125 I | op-osd: OSD orchestration status for PVC smithi179-lv-3-data-0mvm5g is "starting" 2022-02-21T18:38:11.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:11.061138 I | op-osd: OSD orchestration status for PVC smithi179-lv-4-data-0thzk9 is "starting" 2022-02-21T18:38:12.463 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:13.607 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:13.627 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T18:38:13.628 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:38:23.629 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:25.212 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:25.262 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T18:38:25.263 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:38:26.560 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:26.581067 I | op-osd: OSD orchestration status for node smithi140-lv-1-data-0sfl8x is "orchestrating" 2022-02-21T18:38:28.345 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:28.365434 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0sfl8x is "orchestrating" 2022-02-21T18:38:28.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:28.893128 I | op-osd: OSD orchestration status for node smithi179-lv-1-data-0gqt5r is "orchestrating" 2022-02-21T18:38:28.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:28.893205 I | op-osd: OSD orchestration status for node smithi179-lv-2-data-0bk9j2 is "orchestrating" 2022-02-21T18:38:29.393 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:29.414100 I | op-osd: OSD orchestration status for node smithi140-lv-2-data-0s5fp2 is "orchestrating" 2022-02-21T18:38:30.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.058668 I | op-osd: OSD orchestration status for PVC smithi179-lv-2-data-0bk9j2 is "orchestrating" 2022-02-21T18:38:30.041 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.059298 I | op-osd: OSD orchestration status for PVC smithi179-lv-1-data-0gqt5r is "orchestrating" 2022-02-21T18:38:30.380 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.401294 I | op-osd: OSD orchestration status for node smithi081-lv-3-data-0tvdqv is "orchestrating" 2022-02-21T18:38:30.557 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.577995 I | op-osd: OSD orchestration status for PVC smithi140-lv-2-data-0s5fp2 is "orchestrating" 2022-02-21T18:38:30.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.611955 I | op-osd: OSD orchestration status for node smithi179-lv-4-data-0thzk9 is "orchestrating" 2022-02-21T18:38:30.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.795283 I | op-osd: OSD orchestration status for node smithi140-lv-3-data-07nf5l is "orchestrating" 2022-02-21T18:38:30.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.795499 I | op-osd: OSD orchestration status for node smithi140-lv-4-data-05zd97 is "orchestrating" 2022-02-21T18:38:30.880 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.901327 I | op-osd: OSD orchestration status for node smithi081-lv-4-data-0tqs4h is "orchestrating" 2022-02-21T18:38:30.919 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:30.940968 I | op-osd: OSD orchestration status for PVC smithi081-lv-3-data-0tvdqv is "orchestrating" 2022-02-21T18:38:31.290 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:31.310836 I | op-osd: OSD orchestration status for PVC smithi140-lv-3-data-07nf5l is "orchestrating" 2022-02-21T18:38:31.803 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:31.824577 I | op-osd: OSD orchestration status for PVC smithi140-lv-4-data-05zd97 is "orchestrating" 2022-02-21T18:38:31.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:31.824612 I | op-osd: OSD orchestration status for PVC smithi179-lv-4-data-0thzk9 is "orchestrating" 2022-02-21T18:38:31.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:31.824726 I | op-osd: OSD orchestration status for PVC smithi081-lv-4-data-0tqs4h is "orchestrating" 2022-02-21T18:38:33.121 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:33.142584 I | op-osd: OSD orchestration status for node smithi179-lv-3-data-0mvm5g is "orchestrating" 2022-02-21T18:38:33.709 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:33.730842 I | op-osd: OSD orchestration status for PVC smithi179-lv-3-data-0mvm5g is "orchestrating" 2022-02-21T18:38:35.264 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:35.353 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:35.375133 I | op-osd: OSD orchestration status for node smithi081-lv-2-data-08pkzc is "orchestrating" 2022-02-21T18:38:36.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:36.217518 I | op-osd: OSD orchestration status for PVC smithi081-lv-2-data-08pkzc is "orchestrating" 2022-02-21T18:38:36.197 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:36.217779 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0sfl8x is "completed" 2022-02-21T18:38:36.197 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:36.217821 I | op-osd: creating OSD 0 on PVC "smithi140-lv-1-data-0sfl8x" 2022-02-21T18:38:36.197 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:36.217842 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0sfl8x" 2022-02-21T18:38:36.222 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:36.252 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":9,"num_osds":7,"num_up_osds":0,"osd_up_since":0,"num_in_osds":7,"osd_in_since":1645468715,"num_remapped_pgs":0} 2022-02-21T18:38:36.254 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:38:37.347 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:37.366549 I | op-osd: OSD orchestration status for node smithi081-lv-1-data-0zzmvw is "orchestrating" 2022-02-21T18:38:37.715 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:37.735227 I | op-osd: OSD orchestration status for PVC smithi081-lv-1-data-0zzmvw is "orchestrating" 2022-02-21T18:38:38.178 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:38.199274 I | op-osd: OSD orchestration status for PVC smithi179-lv-2-data-0bk9j2 is "completed" 2022-02-21T18:38:38.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:38.199311 I | op-osd: creating OSD 2 on PVC "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:38:38.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:38.199327 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:38:39.854 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:39.875011 I | op-osd: OSD orchestration status for PVC smithi140-lv-2-data-0s5fp2 is "completed" 2022-02-21T18:38:39.854 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:39.875042 I | op-osd: creating OSD 1 on PVC "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:38:39.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:39.875056 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:38:39.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.008159 I | op-osd: OSD orchestration status for PVC smithi179-lv-1-data-0gqt5r is "completed" 2022-02-21T18:38:39.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.008198 I | op-osd: creating OSD 3 on PVC "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:38:39.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.008222 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:38:40.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.359796 I | op-osd: OSD orchestration status for PVC smithi179-lv-4-data-0thzk9 is "completed" 2022-02-21T18:38:40.339 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.359832 I | op-osd: creating OSD 5 on PVC "smithi179-lv-4-data-0thzk9" 2022-02-21T18:38:40.339 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.359850 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:38:40.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.998272 I | op-osd: OSD orchestration status for PVC smithi140-lv-3-data-07nf5l is "completed" 2022-02-21T18:38:40.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.998302 I | op-osd: creating OSD 4 on PVC "smithi140-lv-3-data-07nf5l" 2022-02-21T18:38:40.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:40.998316 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:38:41.693 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:41.714754 I | op-osd: OSD orchestration status for PVC smithi140-lv-4-data-05zd97 is "completed" 2022-02-21T18:38:41.694 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:41.714782 I | op-osd: creating OSD 6 on PVC "smithi140-lv-4-data-05zd97" 2022-02-21T18:38:41.694 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:41.714794 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:38:42.381 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:42.402520 I | op-osd: OSD orchestration status for PVC smithi081-lv-3-data-0tvdqv is "completed" 2022-02-21T18:38:42.382 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:42.402561 I | op-osd: creating OSD 7 on PVC "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:38:42.382 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:42.402582 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:38:43.630 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:43.649187 I | op-osd: OSD orchestration status for PVC smithi081-lv-4-data-0tqs4h is "completed" 2022-02-21T18:38:43.630 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:43.649208 I | op-osd: creating OSD 8 on PVC "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:38:43.631 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:43.649215 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:38:43.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:43.962292 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi081": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:38:44.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.078157 I | op-osd: OSD orchestration status for PVC smithi179-lv-3-data-0mvm5g is "completed" 2022-02-21T18:38:44.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.078180 I | op-osd: creating OSD 9 on PVC "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:38:44.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.078191 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:38:44.743 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.764623 I | op-osd: OSD orchestration status for PVC smithi081-lv-2-data-08pkzc is "completed" 2022-02-21T18:38:44.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.764655 I | op-osd: creating OSD 10 on PVC "smithi081-lv-2-data-08pkzc" 2022-02-21T18:38:44.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:44.764666 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:38:46.255 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:47.169 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:47.186 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":0} 2022-02-21T18:38:47.186 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:38:48.087 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:48.108425 I | op-osd: OSD orchestration status for PVC smithi081-lv-1-data-0zzmvw is "completed" 2022-02-21T18:38:48.088 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:48.108463 I | op-osd: creating OSD 11 on PVC "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:38:48.089 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:48.108478 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:38:51.439 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:51.460643 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T18:38:51.440 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:38:51.460663 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:38:57.187 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:38:59.598 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:38:59.622 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":0} 2022-02-21T18:38:59.623 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:39:09.623 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:39:11.851 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:11.868 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":0} 2022-02-21T18:39:11.868 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T18:39:21.870 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:39:22.882 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:22.905 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"num_osds":12,"num_up_osds":5,"osd_up_since":1645468762,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":1} 2022-02-21T18:39:22.906 INFO:tasks.rook: have 5/12 OSDs 2022-02-21T18:39:32.908 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:39:33.477 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:33.496 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"num_osds":12,"num_up_osds":8,"osd_up_since":1645468765,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":0} 2022-02-21T18:39:33.496 INFO:tasks.rook: have 8/12 OSDs 2022-02-21T18:39:43.498 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd stat -f json 2022-02-21T18:39:44.059 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:44.077 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"num_osds":12,"num_up_osds":12,"osd_up_since":1645468778,"num_in_osds":12,"osd_in_since":1645468719,"num_remapped_pgs":0} 2022-02-21T18:39:44.077 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-02-21T18:39:44.078 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- cat /etc/ceph/ceph.conf 2022-02-21T18:39:44.272 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2022-02-21T18:39:44.273 INFO:teuthology.orchestra.run.smithi081.stdout:mon_host = 10.105.216.166:6789,10.105.182.125:6789,10.104.249.171:6789 2022-02-21T18:39:44.273 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:44.273 INFO:teuthology.orchestra.run.smithi081.stdout:[client.admin] 2022-02-21T18:39:44.273 INFO:teuthology.orchestra.run.smithi081.stdout:keyring = /etc/ceph/keyring 2022-02-21T18:39:44.277 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- cat /etc/ceph/keyring 2022-02-21T18:39:44.527 INFO:teuthology.orchestra.run.smithi081.stdout:[client.admin] 2022-02-21T18:39:44.528 INFO:teuthology.orchestra.run.smithi081.stdout:key = AQB+2hNiXeZpFxAAlSaWTLg1LLo+IKHwgIdzsg== 2022-02-21T18:39:44.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2022-02-21T18:39:44.546 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /etc/ceph 2022-02-21T18:39:44.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2022-02-21T18:39:44.579 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:39:44.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T18:39:44.590 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:39:44.591 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/keyring 2022-02-21T18:39:44.649 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:39:44.649 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T18:39:44.664 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:39:44.665 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/keyring 2022-02-21T18:39:44.679 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:39:44.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T18:39:44.694 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:39:44.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/keyring 2022-02-21T18:39:44.755 INFO:tasks.rook:Setting up client nodes... 2022-02-21T18:39:44.756 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T18:39:45.534 INFO:teuthology.orchestra.run.smithi081.stdout:[client.a] 2022-02-21T18:39:45.534 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBx3BNiAuFQGhAAJDICLT0oNpxEq/E163hXbw== 2022-02-21T18:39:45.558 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-02-21T18:39:45.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-02-21T18:39:45.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-02-21T18:39:45.576 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T18:39:46.333 INFO:teuthology.orchestra.run.smithi081.stdout:[client.b] 2022-02-21T18:39:46.333 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBy3BNioHS5DRAAJ9BOxxaJjZjLKMHh+U6Ozw== 2022-02-21T18:39:46.357 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2022-02-21T18:39:46.357 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-02-21T18:39:46.357 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-02-21T18:39:46.376 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T18:39:47.189 INFO:teuthology.orchestra.run.smithi081.stdout:[client.c] 2022-02-21T18:39:47.190 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBz3BNiXkO0AxAA12Qz9rQvCWwZHqbKKWwN3w== 2022-02-21T18:39:47.212 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-02-21T18:39:47.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-02-21T18:39:47.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-02-21T18:39:47.236 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd dump --format=json 2022-02-21T18:39:47.749 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:47.749 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"fsid":"5726039d-30b1-4825-8940-502857d33853","created":"2022-02-21T18:31:51.538917+0000","modified":"2022-02-21T18:39:39.115472+0000","last_up_change":"2022-02-21T18:39:38.006256+0000","last_in_change":"2022-02-21T18:38:39.703637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T18:39:17.946971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"958866b3-4e7b-49e8-9859-7a85ab4385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6800","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6801","nonce":4256015876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6802","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6803","nonce":4256015876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6806","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6807","nonce":4256015876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6804","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6805","nonce":4256015876}]},"public_addr":"10.250.135.15:6801/4256015876","cluster_addr":"10.250.135.15:6803/4256015876","heartbeat_back_addr":"10.250.135.15:6807/4256015876","heartbeat_front_addr":"10.250.135.15:6805/4256015876","state":["exists","up"]},{"osd":1,"uuid":"233409cd-f88f-46bb-8f2e-6bc8b87166dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6800","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6801","nonce":1491236289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6802","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6803","nonce":1491236289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6806","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6807","nonce":1491236289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6804","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6805","nonce":1491236289}]},"public_addr":"10.250.135.16:6801/1491236289","cluster_addr":"10.250.135.16:6803/1491236289","heartbeat_back_addr":"10.250.135.16:6807/1491236289","heartbeat_front_addr":"10.250.135.16:6805/1491236289","state":["exists","up"]},{"osd":2,"uuid":"d2ac45bc-a186-4e51-86b7-3a3ec5caba73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6800","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6801","nonce":1470469769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6802","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6803","nonce":1470469769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6806","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6807","nonce":1470469769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6804","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6805","nonce":1470469769}]},"public_addr":"10.250.134.77:6801/1470469769","cluster_addr":"10.250.134.77:6803/1470469769","heartbeat_back_addr":"10.250.134.77:6807/1470469769","heartbeat_front_addr":"10.250.134.77:6805/1470469769","state":["exists","up"]},{"osd":3,"uuid":"d38a78a9-3cd3-4051-8b81-a27b18ab0c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6800","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6801","nonce":587882047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6802","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6803","nonce":587882047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6806","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6807","nonce":587882047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6804","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6805","nonce":587882047}]},"public_addr":"10.250.134.78:6801/587882047","cluster_addr":"10.250.134.78:6803/587882047","heartbeat_back_addr":"10.250.134.78:6807/587882047","heartbeat_front_addr":"10.250.134.78:6805/587882047","state":["exists","up"]},{"osd":4,"uuid":"610dfd57-4faa-4d9b-801f-901411ff8492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6800","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6801","nonce":801250050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6802","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6803","nonce":801250050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6806","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6807","nonce":801250050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6804","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6805","nonce":801250050}]},"public_addr":"10.250.135.17:6801/801250050","cluster_addr":"10.250.135.17:6803/801250050","heartbeat_back_addr":"10.250.135.17:6807/801250050","heartbeat_front_addr":"10.250.135.17:6805/801250050","state":["exists","up"]},{"osd":5,"uuid":"fe218eb0-609d-4b1a-a344-cf5f4e6caf3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6800","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6801","nonce":2054811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6802","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6803","nonce":2054811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6806","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6807","nonce":2054811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6804","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6805","nonce":2054811008}]},"public_addr":"10.250.134.79:6801/2054811008","cluster_addr":"10.250.134.79:6803/2054811008","heartbeat_back_addr":"10.250.134.79:6807/2054811008","heartbeat_front_addr":"10.250.134.79:6805/2054811008","state":["exists","up"]},{"osd":6,"uuid":"50295aff-e026-4ff7-8b8b-c77a783af910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6800","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6801","nonce":1804172581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6802","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6803","nonce":1804172581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6806","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6807","nonce":1804172581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6804","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6805","nonce":1804172581}]},"public_addr":"10.250.135.18:6801/1804172581","cluster_addr":"10.250.135.18:6803/1804172581","heartbeat_back_addr":"10.250.135.18:6807/1804172581","heartbeat_front_addr":"10.250.135.18:6805/1804172581","state":["exists","up"]},{"osd":7,"uuid":"f4be2cce-05cc-4f54-b6d7-d608e49fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6800","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6801","nonce":1678918836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6802","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6803","nonce":1678918836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6806","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6807","nonce":1678918836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6804","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6805","nonce":1678918836}]},"public_addr":"10.250.131.139:6801/1678918836","cluster_addr":"10.250.131.139:6803/1678918836","heartbeat_back_addr":"10.250.131.139:6807/1678918836","heartbeat_front_addr":"10.250.131.139:6805/1678918836","state":["exists","up"]},{"osd":8,"uuid":"819b23f0-83f9-4eb8-9c1a-e28580967c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6800","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6801","nonce":1479644580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6802","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6803","nonce":1479644580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6806","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6807","nonce":1479644580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6804","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6805","nonce":1479644580}]},"public_addr":"10.250.131.141:6801/1479644580","cluster_addr":"10.250.131.141:6803/1479644580","heartbeat_back_addr":"10.250.131.141:6807/1479644580","heartbeat_front_addr":"10.250.131.141:6805/1479644580","state":["exists","up"]},{"osd":9,"uuid":"522ff4c0-4b99-47f7-b2f1-1d323a9eddfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6800","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6801","nonce":2465120023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6802","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6803","nonce":2465120023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6806","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6807","nonce":2465120023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6804","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6805","nonce":2465120023}]},"public_addr":"10.250.134.80:6801/2465120023","cluster_addr":"10.250.134.80:6803/2465120023","heartbeat_back_addr":"10.250.134.80:6807/2465120023","heartbeat_front_addr":"10.250.134.80:6805/2465120023","state":["exists","up"]},{"osd":10,"uuid":"3f6caa13-0310-4aed-88e6-8739ce4b2ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6800","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6801","nonce":872474064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6802","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6803","nonce":872474064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6806","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6807","nonce":872474064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6804","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6805","nonce":872474064}]},"public_addr":"10.250.131.142:6801/872474064","cluster_addr":"10.250.131.142:6803/872474064","heartbeat_back_addr":"10.250.131.142:6807/872474064","heartbeat_front_addr":"10.250.131.142:6805/872474064","state":["exists","up"]},{"osd":11,"uuid":"c49a5bcb-dec1-4748-b6f5-913e5e0e5d0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6800","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6801","nonce":2841677057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6802","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6803","nonce":2841677057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6806","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6807","nonce":2841677057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6804","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6805","nonce":2841677057}]},"public_addr":"10.250.131.143:6801/2841677057","cluster_addr":"10.250.131.143:6803/2841677057","heartbeat_back_addr":"10.250.131.143:6807/2841677057","heartbeat_front_addr":"10.250.131.143:6805/2841677057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:05.079010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:10.329210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:11.928536+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:18.688716+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.746453+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375291+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.671828+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.113430+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.643934+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:36.309515+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.433565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.131.133:0/2857517469":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/2549655":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6801/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6800/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/677704742":"2022-02-22T18:37:27.123953+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T18:39:47.767 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-02-21T18:39:17.946971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-02-21T18:39:47.768 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd pool get .mgr pg_num 2022-02-21T18:39:48.268 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-02-21T18:39:48.287 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-02-21T18:39:48.287 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-02-21T18:39:48.288 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph mgr dump --format=json 2022-02-21T18:39:48.894 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:48.912 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":13,"active_gid":14257,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.133:6800","nonce":4034332667},{"type":"v1","addr":"10.250.131.133:6801","nonce":4034332667}]},"active_addr":"10.250.131.133:6801/4034332667","active_change":"2022-02-21T18:37:27.158940+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.250.131.133:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.250.131.133:0","nonce":3192497543}]},{"addrvec":[{"type":"v2","addr":"10.250.131.133:0","nonce":2801708849}]},{"addrvec":[{"type":"v2","addr":"10.250.131.133:0","nonce":1270226762}]},{"addrvec":[{"type":"v2","addr":"10.250.131.133:0","nonce":2969610038}]}]}} 2022-02-21T18:39:48.916 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-02-21T18:39:48.917 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-02-21T18:39:48.917 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd dump --format=json 2022-02-21T18:39:49.438 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:49.439 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"fsid":"5726039d-30b1-4825-8940-502857d33853","created":"2022-02-21T18:31:51.538917+0000","modified":"2022-02-21T18:39:39.115472+0000","last_up_change":"2022-02-21T18:39:38.006256+0000","last_in_change":"2022-02-21T18:38:39.703637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T18:39:17.946971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"958866b3-4e7b-49e8-9859-7a85ab4385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6800","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6801","nonce":4256015876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6802","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6803","nonce":4256015876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6806","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6807","nonce":4256015876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6804","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6805","nonce":4256015876}]},"public_addr":"10.250.135.15:6801/4256015876","cluster_addr":"10.250.135.15:6803/4256015876","heartbeat_back_addr":"10.250.135.15:6807/4256015876","heartbeat_front_addr":"10.250.135.15:6805/4256015876","state":["exists","up"]},{"osd":1,"uuid":"233409cd-f88f-46bb-8f2e-6bc8b87166dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6800","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6801","nonce":1491236289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6802","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6803","nonce":1491236289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6806","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6807","nonce":1491236289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6804","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6805","nonce":1491236289}]},"public_addr":"10.250.135.16:6801/1491236289","cluster_addr":"10.250.135.16:6803/1491236289","heartbeat_back_addr":"10.250.135.16:6807/1491236289","heartbeat_front_addr":"10.250.135.16:6805/1491236289","state":["exists","up"]},{"osd":2,"uuid":"d2ac45bc-a186-4e51-86b7-3a3ec5caba73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6800","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6801","nonce":1470469769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6802","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6803","nonce":1470469769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6806","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6807","nonce":1470469769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6804","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6805","nonce":1470469769}]},"public_addr":"10.250.134.77:6801/1470469769","cluster_addr":"10.250.134.77:6803/1470469769","heartbeat_back_addr":"10.250.134.77:6807/1470469769","heartbeat_front_addr":"10.250.134.77:6805/1470469769","state":["exists","up"]},{"osd":3,"uuid":"d38a78a9-3cd3-4051-8b81-a27b18ab0c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6800","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6801","nonce":587882047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6802","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6803","nonce":587882047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6806","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6807","nonce":587882047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6804","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6805","nonce":587882047}]},"public_addr":"10.250.134.78:6801/587882047","cluster_addr":"10.250.134.78:6803/587882047","heartbeat_back_addr":"10.250.134.78:6807/587882047","heartbeat_front_addr":"10.250.134.78:6805/587882047","state":["exists","up"]},{"osd":4,"uuid":"610dfd57-4faa-4d9b-801f-901411ff8492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6800","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6801","nonce":801250050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6802","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6803","nonce":801250050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6806","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6807","nonce":801250050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6804","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6805","nonce":801250050}]},"public_addr":"10.250.135.17:6801/801250050","cluster_addr":"10.250.135.17:6803/801250050","heartbeat_back_addr":"10.250.135.17:6807/801250050","heartbeat_front_addr":"10.250.135.17:6805/801250050","state":["exists","up"]},{"osd":5,"uuid":"fe218eb0-609d-4b1a-a344-cf5f4e6caf3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6800","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6801","nonce":2054811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6802","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6803","nonce":2054811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6806","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6807","nonce":2054811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6804","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6805","nonce":2054811008}]},"public_addr":"10.250.134.79:6801/2054811008","cluster_addr":"10.250.134.79:6803/2054811008","heartbeat_back_addr":"10.250.134.79:6807/2054811008","heartbeat_front_addr":"10.250.134.79:6805/2054811008","state":["exists","up"]},{"osd":6,"uuid":"50295aff-e026-4ff7-8b8b-c77a783af910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6800","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6801","nonce":1804172581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6802","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6803","nonce":1804172581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6806","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6807","nonce":1804172581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6804","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6805","nonce":1804172581}]},"public_addr":"10.250.135.18:6801/1804172581","cluster_addr":"10.250.135.18:6803/1804172581","heartbeat_back_addr":"10.250.135.18:6807/1804172581","heartbeat_front_addr":"10.250.135.18:6805/1804172581","state":["exists","up"]},{"osd":7,"uuid":"f4be2cce-05cc-4f54-b6d7-d608e49fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6800","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6801","nonce":1678918836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6802","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6803","nonce":1678918836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6806","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6807","nonce":1678918836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6804","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6805","nonce":1678918836}]},"public_addr":"10.250.131.139:6801/1678918836","cluster_addr":"10.250.131.139:6803/1678918836","heartbeat_back_addr":"10.250.131.139:6807/1678918836","heartbeat_front_addr":"10.250.131.139:6805/1678918836","state":["exists","up"]},{"osd":8,"uuid":"819b23f0-83f9-4eb8-9c1a-e28580967c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6800","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6801","nonce":1479644580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6802","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6803","nonce":1479644580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6806","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6807","nonce":1479644580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6804","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6805","nonce":1479644580}]},"public_addr":"10.250.131.141:6801/1479644580","cluster_addr":"10.250.131.141:6803/1479644580","heartbeat_back_addr":"10.250.131.141:6807/1479644580","heartbeat_front_addr":"10.250.131.141:6805/1479644580","state":["exists","up"]},{"osd":9,"uuid":"522ff4c0-4b99-47f7-b2f1-1d323a9eddfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6800","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6801","nonce":2465120023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6802","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6803","nonce":2465120023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6806","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6807","nonce":2465120023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6804","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6805","nonce":2465120023}]},"public_addr":"10.250.134.80:6801/2465120023","cluster_addr":"10.250.134.80:6803/2465120023","heartbeat_back_addr":"10.250.134.80:6807/2465120023","heartbeat_front_addr":"10.250.134.80:6805/2465120023","state":["exists","up"]},{"osd":10,"uuid":"3f6caa13-0310-4aed-88e6-8739ce4b2ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6800","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6801","nonce":872474064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6802","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6803","nonce":872474064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6806","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6807","nonce":872474064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6804","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6805","nonce":872474064}]},"public_addr":"10.250.131.142:6801/872474064","cluster_addr":"10.250.131.142:6803/872474064","heartbeat_back_addr":"10.250.131.142:6807/872474064","heartbeat_front_addr":"10.250.131.142:6805/872474064","state":["exists","up"]},{"osd":11,"uuid":"c49a5bcb-dec1-4748-b6f5-913e5e0e5d0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6800","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6801","nonce":2841677057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6802","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6803","nonce":2841677057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6806","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6807","nonce":2841677057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6804","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6805","nonce":2841677057}]},"public_addr":"10.250.131.143:6801/2841677057","cluster_addr":"10.250.131.143:6803/2841677057","heartbeat_back_addr":"10.250.131.143:6807/2841677057","heartbeat_front_addr":"10.250.131.143:6805/2841677057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:05.079010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:10.329210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:11.928536+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:18.688716+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.746453+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375291+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.671828+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.113430+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.643934+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:36.309515+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.433565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.131.133:0/2857517469":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/2549655":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6801/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6800/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/677704742":"2022-02-22T18:37:27.123953+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T18:39:49.460 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-02-21T18:39:49.460 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd dump --format=json 2022-02-21T18:39:50.042 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:50.042 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"fsid":"5726039d-30b1-4825-8940-502857d33853","created":"2022-02-21T18:31:51.538917+0000","modified":"2022-02-21T18:39:39.115472+0000","last_up_change":"2022-02-21T18:39:38.006256+0000","last_in_change":"2022-02-21T18:38:39.703637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T18:39:17.946971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"958866b3-4e7b-49e8-9859-7a85ab4385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6800","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6801","nonce":4256015876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6802","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6803","nonce":4256015876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6806","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6807","nonce":4256015876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.15:6804","nonce":4256015876},{"type":"v1","addr":"10.250.135.15:6805","nonce":4256015876}]},"public_addr":"10.250.135.15:6801/4256015876","cluster_addr":"10.250.135.15:6803/4256015876","heartbeat_back_addr":"10.250.135.15:6807/4256015876","heartbeat_front_addr":"10.250.135.15:6805/4256015876","state":["exists","up"]},{"osd":1,"uuid":"233409cd-f88f-46bb-8f2e-6bc8b87166dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6800","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6801","nonce":1491236289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6802","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6803","nonce":1491236289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6806","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6807","nonce":1491236289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.16:6804","nonce":1491236289},{"type":"v1","addr":"10.250.135.16:6805","nonce":1491236289}]},"public_addr":"10.250.135.16:6801/1491236289","cluster_addr":"10.250.135.16:6803/1491236289","heartbeat_back_addr":"10.250.135.16:6807/1491236289","heartbeat_front_addr":"10.250.135.16:6805/1491236289","state":["exists","up"]},{"osd":2,"uuid":"d2ac45bc-a186-4e51-86b7-3a3ec5caba73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6800","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6801","nonce":1470469769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6802","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6803","nonce":1470469769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6806","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6807","nonce":1470469769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.77:6804","nonce":1470469769},{"type":"v1","addr":"10.250.134.77:6805","nonce":1470469769}]},"public_addr":"10.250.134.77:6801/1470469769","cluster_addr":"10.250.134.77:6803/1470469769","heartbeat_back_addr":"10.250.134.77:6807/1470469769","heartbeat_front_addr":"10.250.134.77:6805/1470469769","state":["exists","up"]},{"osd":3,"uuid":"d38a78a9-3cd3-4051-8b81-a27b18ab0c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6800","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6801","nonce":587882047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6802","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6803","nonce":587882047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6806","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6807","nonce":587882047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.78:6804","nonce":587882047},{"type":"v1","addr":"10.250.134.78:6805","nonce":587882047}]},"public_addr":"10.250.134.78:6801/587882047","cluster_addr":"10.250.134.78:6803/587882047","heartbeat_back_addr":"10.250.134.78:6807/587882047","heartbeat_front_addr":"10.250.134.78:6805/587882047","state":["exists","up"]},{"osd":4,"uuid":"610dfd57-4faa-4d9b-801f-901411ff8492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6800","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6801","nonce":801250050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6802","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6803","nonce":801250050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6806","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6807","nonce":801250050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.17:6804","nonce":801250050},{"type":"v1","addr":"10.250.135.17:6805","nonce":801250050}]},"public_addr":"10.250.135.17:6801/801250050","cluster_addr":"10.250.135.17:6803/801250050","heartbeat_back_addr":"10.250.135.17:6807/801250050","heartbeat_front_addr":"10.250.135.17:6805/801250050","state":["exists","up"]},{"osd":5,"uuid":"fe218eb0-609d-4b1a-a344-cf5f4e6caf3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6800","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6801","nonce":2054811008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6802","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6803","nonce":2054811008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6806","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6807","nonce":2054811008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.79:6804","nonce":2054811008},{"type":"v1","addr":"10.250.134.79:6805","nonce":2054811008}]},"public_addr":"10.250.134.79:6801/2054811008","cluster_addr":"10.250.134.79:6803/2054811008","heartbeat_back_addr":"10.250.134.79:6807/2054811008","heartbeat_front_addr":"10.250.134.79:6805/2054811008","state":["exists","up"]},{"osd":6,"uuid":"50295aff-e026-4ff7-8b8b-c77a783af910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6800","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6801","nonce":1804172581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6802","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6803","nonce":1804172581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6806","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6807","nonce":1804172581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.135.18:6804","nonce":1804172581},{"type":"v1","addr":"10.250.135.18:6805","nonce":1804172581}]},"public_addr":"10.250.135.18:6801/1804172581","cluster_addr":"10.250.135.18:6803/1804172581","heartbeat_back_addr":"10.250.135.18:6807/1804172581","heartbeat_front_addr":"10.250.135.18:6805/1804172581","state":["exists","up"]},{"osd":7,"uuid":"f4be2cce-05cc-4f54-b6d7-d608e49fd3e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6800","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6801","nonce":1678918836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6802","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6803","nonce":1678918836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6806","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6807","nonce":1678918836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.139:6804","nonce":1678918836},{"type":"v1","addr":"10.250.131.139:6805","nonce":1678918836}]},"public_addr":"10.250.131.139:6801/1678918836","cluster_addr":"10.250.131.139:6803/1678918836","heartbeat_back_addr":"10.250.131.139:6807/1678918836","heartbeat_front_addr":"10.250.131.139:6805/1678918836","state":["exists","up"]},{"osd":8,"uuid":"819b23f0-83f9-4eb8-9c1a-e28580967c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6800","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6801","nonce":1479644580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6802","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6803","nonce":1479644580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6806","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6807","nonce":1479644580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.141:6804","nonce":1479644580},{"type":"v1","addr":"10.250.131.141:6805","nonce":1479644580}]},"public_addr":"10.250.131.141:6801/1479644580","cluster_addr":"10.250.131.141:6803/1479644580","heartbeat_back_addr":"10.250.131.141:6807/1479644580","heartbeat_front_addr":"10.250.131.141:6805/1479644580","state":["exists","up"]},{"osd":9,"uuid":"522ff4c0-4b99-47f7-b2f1-1d323a9eddfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6800","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6801","nonce":2465120023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6802","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6803","nonce":2465120023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6806","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6807","nonce":2465120023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.134.80:6804","nonce":2465120023},{"type":"v1","addr":"10.250.134.80:6805","nonce":2465120023}]},"public_addr":"10.250.134.80:6801/2465120023","cluster_addr":"10.250.134.80:6803/2465120023","heartbeat_back_addr":"10.250.134.80:6807/2465120023","heartbeat_front_addr":"10.250.134.80:6805/2465120023","state":["exists","up"]},{"osd":10,"uuid":"3f6caa13-0310-4aed-88e6-8739ce4b2ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6800","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6801","nonce":872474064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6802","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6803","nonce":872474064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6806","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6807","nonce":872474064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.142:6804","nonce":872474064},{"type":"v1","addr":"10.250.131.142:6805","nonce":872474064}]},"public_addr":"10.250.131.142:6801/872474064","cluster_addr":"10.250.131.142:6803/872474064","heartbeat_back_addr":"10.250.131.142:6807/872474064","heartbeat_front_addr":"10.250.131.142:6805/872474064","state":["exists","up"]},{"osd":11,"uuid":"c49a5bcb-dec1-4748-b6f5-913e5e0e5d0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6800","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6801","nonce":2841677057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6802","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6803","nonce":2841677057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6806","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6807","nonce":2841677057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.131.143:6804","nonce":2841677057},{"type":"v1","addr":"10.250.131.143:6805","nonce":2841677057}]},"public_addr":"10.250.131.143:6801/2841677057","cluster_addr":"10.250.131.143:6803/2841677057","heartbeat_back_addr":"10.250.131.143:6807/2841677057","heartbeat_front_addr":"10.250.131.143:6805/2841677057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:05.079010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:10.329210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:11.928536+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:18.688716+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.746453+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375291+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:19.671828+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.113430+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.643934+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:20.375285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:36.309515+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T18:39:33.433565+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.131.133:0/2857517469":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/2549655":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6801/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:6800/2611815540":"2022-02-22T18:37:27.123953+0000","10.250.131.133:0/677704742":"2022-02-22T18:37:27.123953+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T18:39:50.076 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.0 flush_pg_stats 2022-02-21T18:39:50.076 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.1 flush_pg_stats 2022-02-21T18:39:50.077 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.2 flush_pg_stats 2022-02-21T18:39:50.078 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.3 flush_pg_stats 2022-02-21T18:39:50.078 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.4 flush_pg_stats 2022-02-21T18:39:50.079 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.5 flush_pg_stats 2022-02-21T18:39:50.080 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.6 flush_pg_stats 2022-02-21T18:39:50.080 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.7 flush_pg_stats 2022-02-21T18:39:50.081 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.8 flush_pg_stats 2022-02-21T18:39:50.082 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.9 flush_pg_stats 2022-02-21T18:39:50.082 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.10 flush_pg_stats 2022-02-21T18:39:50.083 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph tell osd.11 flush_pg_stats 2022-02-21T18:39:50.619 INFO:teuthology.orchestra.run.smithi081.stdout:98784247818 2022-02-21T18:39:50.619 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.4 2022-02-21T18:39:50.904 INFO:teuthology.orchestra.run.smithi081.stdout:107374182408 2022-02-21T18:39:50.904 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.5 2022-02-21T18:39:50.907 INFO:teuthology.orchestra.run.smithi081.stdout:73014444043 2022-02-21T18:39:50.908 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.0 2022-02-21T18:39:50.911 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:50.911 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.7 2022-02-21T18:39:50.916 INFO:teuthology.orchestra.run.smithi081.stdout:141733920772 2022-02-21T18:39:50.917 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.10 2022-02-21T18:39:50.919 INFO:teuthology.orchestra.run.smithi081.stdout:98784247817 2022-02-21T18:39:50.919 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.3 2022-02-21T18:39:50.921 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:50.921 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.11 2022-02-21T18:39:50.927 INFO:teuthology.orchestra.run.smithi081.stdout:77309411338 2022-02-21T18:39:50.928 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.1 2022-02-21T18:39:50.949 INFO:teuthology.orchestra.run.smithi081.stdout:103079215111 2022-02-21T18:39:50.949 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.9 2022-02-21T18:39:50.982 INFO:teuthology.orchestra.run.smithi081.stdout:81604378633 2022-02-21T18:39:50.983 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.2 2022-02-21T18:39:51.023 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:51.024 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.8 2022-02-21T18:39:51.115 INFO:teuthology.orchestra.run.smithi081.stdout:107374182409 2022-02-21T18:39:51.115 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.6 2022-02-21T18:39:51.354 INFO:teuthology.orchestra.run.smithi081.stdout:98784247816 2022-02-21T18:39:51.581 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247818 got 98784247816 for osd.4 2022-02-21T18:39:51.713 INFO:teuthology.orchestra.run.smithi081.stdout:128849018884 2022-02-21T18:39:51.764 INFO:teuthology.orchestra.run.smithi081.stdout:128849018884 2022-02-21T18:39:51.807 INFO:teuthology.orchestra.run.smithi081.stdout:77309411337 2022-02-21T18:39:51.867 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018886 got 128849018884 for osd.11 2022-02-21T18:39:51.871 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018886 got 128849018884 for osd.7 2022-02-21T18:39:51.874 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.1 2022-02-21T18:39:51.924 INFO:teuthology.orchestra.run.smithi081.stdout:103079215109 2022-02-21T18:39:51.960 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215109 for osd.9 2022-02-21T18:39:51.974 INFO:teuthology.orchestra.run.smithi081.stdout:107374182407 2022-02-21T18:39:51.995 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182408 got 107374182407 for osd.5 2022-02-21T18:39:52.033 INFO:teuthology.orchestra.run.smithi081.stdout:98784247815 2022-02-21T18:39:52.059 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247817 got 98784247815 for osd.3 2022-02-21T18:39:52.085 INFO:teuthology.orchestra.run.smithi081.stdout:73014444042 2022-02-21T18:39:52.108 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444043 got 73014444042 for osd.0 2022-02-21T18:39:52.167 INFO:teuthology.orchestra.run.smithi081.stdout:81604378632 2022-02-21T18:39:52.188 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378633 got 81604378632 for osd.2 2022-02-21T18:39:52.219 INFO:teuthology.orchestra.run.smithi081.stdout:107374182407 2022-02-21T18:39:52.233 INFO:teuthology.orchestra.run.smithi081.stdout:141733920771 2022-02-21T18:39:52.237 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182409 got 107374182407 for osd.6 2022-02-21T18:39:52.250 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920772 got 141733920771 for osd.10 2022-02-21T18:39:52.288 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:52.310 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018886 got 128849018886 for osd.8 2022-02-21T18:39:52.311 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:52.581 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.4 2022-02-21T18:39:52.868 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.11 2022-02-21T18:39:52.873 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.7 2022-02-21T18:39:52.875 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.1 2022-02-21T18:39:52.961 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.9 2022-02-21T18:39:52.996 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.5 2022-02-21T18:39:53.061 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.3 2022-02-21T18:39:53.097 INFO:teuthology.orchestra.run.smithi081.stdout:98784247818 2022-02-21T18:39:53.109 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.0 2022-02-21T18:39:53.161 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247818 got 98784247818 for osd.4 2022-02-21T18:39:53.162 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.189 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.2 2022-02-21T18:39:53.238 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.6 2022-02-21T18:39:53.251 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph osd last-stat-seq osd.10 2022-02-21T18:39:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:53.565 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018886 got 128849018886 for osd.11 2022-02-21T18:39:53.566 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.616 INFO:teuthology.orchestra.run.smithi081.stdout:128849018886 2022-02-21T18:39:53.709 INFO:teuthology.orchestra.run.smithi081.stdout:77309411339 2022-02-21T18:39:53.776 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018886 got 128849018886 for osd.7 2022-02-21T18:39:53.777 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.779 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411338 got 77309411339 for osd.1 2022-02-21T18:39:53.780 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.822 INFO:teuthology.orchestra.run.smithi081.stdout:107374182409 2022-02-21T18:39:53.848 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182408 got 107374182409 for osd.5 2022-02-21T18:39:53.848 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.871 INFO:teuthology.orchestra.run.smithi081.stdout:103079215111 2022-02-21T18:39:53.897 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215111 for osd.9 2022-02-21T18:39:53.897 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:53.929 INFO:teuthology.orchestra.run.smithi081.stdout:73014444043 2022-02-21T18:39:53.952 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444043 got 73014444043 for osd.0 2022-02-21T18:39:53.953 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:54.008 INFO:teuthology.orchestra.run.smithi081.stdout:81604378633 2022-02-21T18:39:54.029 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378633 got 81604378633 for osd.2 2022-02-21T18:39:54.029 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:54.067 INFO:teuthology.orchestra.run.smithi081.stdout:107374182409 2022-02-21T18:39:54.088 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182409 got 107374182409 for osd.6 2022-02-21T18:39:54.088 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:54.092 INFO:teuthology.orchestra.run.smithi081.stdout:98784247817 2022-02-21T18:39:54.134 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247817 got 98784247817 for osd.3 2022-02-21T18:39:54.134 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout:141733920772 2022-02-21T18:39:54.214 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920772 got 141733920772 for osd.10 2022-02-21T18:39:54.214 DEBUG:teuthology.parallel:result is None 2022-02-21T18:39:54.215 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-02-21T18:39:54.216 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph pg dump --format=json 2022-02-21T18:39:54.772 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-02-21T18:39:54.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:54.787 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2022-02-21T18:39:53.459035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66196,"kb_used_data":5492,"kb_used_omap":0,"kb_used_meta":60224,"kb_avail":1124875628,"statfs":{"total":1151940427776,"available":1151872643072,"internally_reserved":0,"allocated":5623808,"data_stored":3300413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61669376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004011"},"pg_stats":[{"pgid":"1.0","version":"34'137","reported_seq":219,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-02-21T18:39:41.151702+0000","last_change":"2022-02-21T18:39:39.235902+0000","last_active":"2022-02-21T18:39:41.151702+0000","last_peered":"2022-02-21T18:39:41.151702+0000","last_clean":"2022-02-21T18:39:41.151702+0000","last_became_active":"2022-02-21T18:39:39.235165+0000","last_became_peered":"2022-02-21T18:39:39.235165+0000","last_unstale":"2022-02-21T18:39:41.151702+0000","last_undegraded":"2022-02-21T18:39:41.151702+0000","last_fullsized":"2022-02-21T18:39:41.151702+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-21T18:39:17.947063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-21T18:39:17.947063+0000","last_clean_scrub_stamp":"2022-02-21T18:39:17.947063+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-22T20:28:53.606631+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,2],"acting":[9,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2054272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":10,"up_from":33,"seq":141733920772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0139999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3039999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6960000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3809999999999998}]}]},{"osd":11,"up_from":30,"seq":128849018886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":864256,"data_stored":670305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":8,"up_from":30,"seq":128849018886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.052}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":7,"up_from":30,"seq":128849018886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.72}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":9,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":868352,"data_stored":671860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2970000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.671}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.786}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9910000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.125}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.323}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1549999999999998}]}]},{"osd":6,"up_from":25,"seq":107374182409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.895}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5264,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739888,"statfs":{"total":95995035648,"available":95989645312,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.863}]}]},{"osd":0,"up_from":17,"seq":73014444043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5712,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739440,"statfs":{"total":95995035648,"available":95989186560,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2570000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.831}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":2,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":868352,"data_stored":671860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]}]},{"osd":3,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.718}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.411}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2999999999999998}]}]},{"osd":4,"up_from":23,"seq":98784247818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0070000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0270000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1549999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3660000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-02-21T18:39:54.789 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph pg dump --format=json 2022-02-21T18:39:55.334 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:55.335 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-02-21T18:39:55.354 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2022-02-21T18:39:53.459035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66196,"kb_used_data":5492,"kb_used_omap":0,"kb_used_meta":60224,"kb_avail":1124875628,"statfs":{"total":1151940427776,"available":1151872643072,"internally_reserved":0,"allocated":5623808,"data_stored":3300413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61669376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004011"},"pg_stats":[{"pgid":"1.0","version":"34'137","reported_seq":219,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-02-21T18:39:41.151702+0000","last_change":"2022-02-21T18:39:39.235902+0000","last_active":"2022-02-21T18:39:41.151702+0000","last_peered":"2022-02-21T18:39:41.151702+0000","last_clean":"2022-02-21T18:39:41.151702+0000","last_became_active":"2022-02-21T18:39:39.235165+0000","last_became_peered":"2022-02-21T18:39:39.235165+0000","last_unstale":"2022-02-21T18:39:41.151702+0000","last_undegraded":"2022-02-21T18:39:41.151702+0000","last_fullsized":"2022-02-21T18:39:41.151702+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-21T18:39:17.947063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-21T18:39:17.947063+0000","last_clean_scrub_stamp":"2022-02-21T18:39:17.947063+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-22T20:28:53.606631+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,2],"acting":[9,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2054272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":10,"up_from":33,"seq":141733920772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0139999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3039999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6960000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3809999999999998}]}]},{"osd":11,"up_from":30,"seq":128849018886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":864256,"data_stored":670305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":8,"up_from":30,"seq":128849018886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.052}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":7,"up_from":30,"seq":128849018886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.72}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":9,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":868352,"data_stored":671860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2970000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.671}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.786}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9910000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.125}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.323}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1549999999999998}]}]},{"osd":6,"up_from":25,"seq":107374182409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.895}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]}]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5264,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739888,"statfs":{"total":95995035648,"available":95989645312,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.863}]}]},{"osd":0,"up_from":17,"seq":73014444043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5712,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93739440,"statfs":{"total":95995035648,"available":95989186560,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2570000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.831}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":2,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":868352,"data_stored":671860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]}]},{"osd":3,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.718}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.411}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3180000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2999999999999998}]}]},{"osd":4,"up_from":23,"seq":98784247818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0070000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9630000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0270000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":335872,"data_stored":142932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1549999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3660000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-02-21T18:39:55.356 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-02-21T18:39:55.356 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-02-21T18:39:55.357 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-02-21T18:39:55.357 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph health --format=json 2022-02-21T18:39:55.896 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:55.896 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-02-21T18:39:55.915 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-02-21T18:39:55.916 INFO:tasks.rook:Rook complete, yielding 2022-02-21T18:39:55.916 INFO:teuthology.run_tasks:Running task rook.shell... 2022-02-21T18:39:55.926 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph -s 2022-02-21T18:39:56.485 INFO:teuthology.orchestra.run.smithi081.stdout: cluster: 2022-02-21T18:39:56.485 INFO:teuthology.orchestra.run.smithi081.stdout: id: 5726039d-30b1-4825-8940-502857d33853 2022-02-21T18:39:56.485 INFO:teuthology.orchestra.run.smithi081.stdout: health: HEALTH_OK 2022-02-21T18:39:56.486 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:56.486 INFO:teuthology.orchestra.run.smithi081.stdout: services: 2022-02-21T18:39:56.486 INFO:teuthology.orchestra.run.smithi081.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-02-21T18:39:56.487 INFO:teuthology.orchestra.run.smithi081.stdout: mgr: a(active, since 2m) 2022-02-21T18:39:56.487 INFO:teuthology.orchestra.run.smithi081.stdout: osd: 12 osds: 12 up (since 18s), 12 in (since 76s) 2022-02-21T18:39:56.487 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:56.487 INFO:teuthology.orchestra.run.smithi081.stdout: data: 2022-02-21T18:39:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: pools: 1 pools, 1 pgs 2022-02-21T18:39:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: objects: 2 objects, 577 KiB 2022-02-21T18:39:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: usage: 65 MiB used, 1.0 TiB / 1.0 TiB avail 2022-02-21T18:39:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: pgs: 1 active+clean 2022-02-21T18:39:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:56.489 INFO:teuthology.orchestra.run.smithi081.stdout: io: 2022-02-21T18:39:56.489 INFO:teuthology.orchestra.run.smithi081.stdout: client: 767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-02-21T18:39:56.489 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:39:56.506 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status 2022-02-21T18:39:57.285 INFO:teuthology.orchestra.run.smithi081.stdout:Backend: rook 2022-02-21T18:39:57.286 INFO:teuthology.orchestra.run.smithi081.stdout:Available: Yes 2022-02-21T18:39:57.308 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ps 2022-02-21T18:39:57.866 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-02-21T18:39:57.866 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi081 running (74s) 0s ago 74s - - f61e120f25ae 2022-02-21T18:39:57.867 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi140 running (2m) 0s ago 2m - - f61e120f25ae 2022-02-21T18:39:57.867 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi179 running (2m) 0s ago 2m - - f61e120f25ae 2022-02-21T18:39:57.867 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.a smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:39:57.867 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a smithi179 running (8m) 0s ago 8m - - f61e120f25ae 2022-02-21T18:39:57.868 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b smithi140 running (7m) 0s ago 7m - - f61e120f25ae 2022-02-21T18:39:57.868 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c smithi081 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T18:39:57.868 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi140 running (81s) 0s ago 81s - - f61e120f25ae 2022-02-21T18:39:57.868 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi140 running (77s) 0s ago 78s - - f61e120f25ae 2022-02-21T18:39:57.869 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 smithi081 running (71s) 0s ago 71s - - f61e120f25ae 2022-02-21T18:39:57.869 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 smithi081 running (67s) 0s ago 67s - - f61e120f25ae 2022-02-21T18:39:57.869 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi179 running (79s) 0s ago 79s - - f61e120f25ae 2022-02-21T18:39:57.869 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi179 running (77s) 0s ago 77s - - f61e120f25ae 2022-02-21T18:39:57.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi140 running (76s) 0s ago 76s - - f61e120f25ae 2022-02-21T18:39:57.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi179 running (77s) 0s ago 77s - - f61e120f25ae 2022-02-21T18:39:57.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi140 running (75s) 0s ago 76s - - f61e120f25ae 2022-02-21T18:39:57.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi081 running (74s) 0s ago 74s - - f61e120f25ae 2022-02-21T18:39:57.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 smithi081 running (73s) 0s ago 74s - - f61e120f25ae 2022-02-21T18:39:57.871 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 smithi179 running (73s) 0s ago 73s - - f61e120f25ae 2022-02-21T18:39:57.889 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls 2022-02-21T18:39:58.571 INFO:teuthology.orchestra.run.smithi081.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-02-21T18:39:58.571 INFO:teuthology.orchestra.run.smithi081.stdout:crash 3/3 0s ago 2m * 2022-02-21T18:39:58.571 INFO:teuthology.orchestra.run.smithi081.stdout:mgr 1/1 0s ago 3m count:1 2022-02-21T18:39:58.572 INFO:teuthology.orchestra.run.smithi081.stdout:mon 3/3 0s ago 8m count:3 2022-02-21T18:39:58.572 INFO:teuthology.orchestra.run.smithi081.stdout:osd 24 0s ago 82s 2022-02-21T18:39:58.572 INFO:teuthology.orchestra.run.smithi081.stdout:osd.all-available-devices 0 0s ago - * 2022-02-21T18:39:58.598 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch device ls 2022-02-21T18:39:59.157 INFO:teuthology.orchestra.run.smithi081.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2022-02-21T18:39:59.157 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:39:59.158 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:39:59.158 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:39:59.158 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:39:59.159 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:39:59.159 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:39:59.159 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:39:59.159 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:39:59.160 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:39:59.160 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:39:59.160 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:39:59.160 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:39:59.178 INFO:teuthology.run_tasks:Running task exec... 2022-02-21T18:39:59.191 INFO:teuthology.task.exec:Executing custom commands... 2022-02-21T18:39:59.192 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi081.front.sepia.ceph.com 2022-02-21T18:39:59.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-02-21T18:39:59.193 DEBUG:teuthology.orchestra.run.smithi081:> toolbox() { 2022-02-21T18:39:59.193 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-02-21T18:39:59.193 DEBUG:teuthology.orchestra.run.smithi081:> } 2022-02-21T18:39:59.194 DEBUG:teuthology.orchestra.run.smithi081:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-02-21T18:39:59.194 DEBUG:teuthology.orchestra.run.smithi081:> toolbox ceph orch osd rm 0 --force 2022-02-21T18:39:59.194 DEBUG:teuthology.orchestra.run.smithi081:> removed_pv="" 2022-02-21T18:39:59.195 DEBUG:teuthology.orchestra.run.smithi081:> while [ "$removed_pv" = "" ] 2022-02-21T18:39:59.195 DEBUG:teuthology.orchestra.run.smithi081:> do 2022-02-21T18:39:59.195 DEBUG:teuthology.orchestra.run.smithi081:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-02-21T18:39:59.196 DEBUG:teuthology.orchestra.run.smithi081:> sleep 3s 2022-02-21T18:39:59.196 DEBUG:teuthology.orchestra.run.smithi081:> done 2022-02-21T18:39:59.196 DEBUG:teuthology.orchestra.run.smithi081:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-02-21T18:39:59.196 DEBUG:teuthology.orchestra.run.smithi081:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-02-21T18:39:59.197 DEBUG:teuthology.orchestra.run.smithi081:> toolbox ceph orch device zap $host $target_path --force 2022-02-21T18:39:59.197 DEBUG:teuthology.orchestra.run.smithi081:> zap_completion="0" 2022-02-21T18:39:59.197 DEBUG:teuthology.orchestra.run.smithi081:> while [ "$zap_completion" = "0" ] 2022-02-21T18:39:59.198 DEBUG:teuthology.orchestra.run.smithi081:> do 2022-02-21T18:39:59.198 DEBUG:teuthology.orchestra.run.smithi081:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-02-21T18:39:59.198 DEBUG:teuthology.orchestra.run.smithi081:> sleep 3s 2022-02-21T18:39:59.199 DEBUG:teuthology.orchestra.run.smithi081:> done 2022-02-21T18:39:59.199 DEBUG:teuthology.orchestra.run.smithi081:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-02-21T18:39:59.200 DEBUG:teuthology.orchestra.run.smithi081:> toolbox ceph orch apply osd --all-available-devices 2022-02-21T18:39:59.200 DEBUG:teuthology.orchestra.run.smithi081:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-02-21T18:39:59.200 DEBUG:teuthology.orchestra.run.smithi081:> num_osd="0" 2022-02-21T18:39:59.201 DEBUG:teuthology.orchestra.run.smithi081:> while [ "$num_osd" != "$orig_num_osd" ] 2022-02-21T18:39:59.201 DEBUG:teuthology.orchestra.run.smithi081:> do 2022-02-21T18:39:59.201 DEBUG:teuthology.orchestra.run.smithi081:> echo "waiting for osd to come back up" 2022-02-21T18:39:59.202 DEBUG:teuthology.orchestra.run.smithi081:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-02-21T18:39:59.202 DEBUG:teuthology.orchestra.run.smithi081:> sleep 30s 2022-02-21T18:39:59.202 DEBUG:teuthology.orchestra.run.smithi081:> done 2022-02-21T18:39:59.203 DEBUG:teuthology.orchestra.run.smithi081:> ' 2022-02-21T18:39:59.219 INFO:teuthology.orchestra.run.smithi081.stderr:++ toolbox ceph osd stat 2022-02-21T18:39:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T18:39:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f3 -d ' ' 2022-02-21T18:40:01.700 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:+ orig_num_osd=12 2022-02-21T18:40:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:+ toolbox ceph orch osd rm 0 --force 2022-02-21T18:40:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-02-21T18:40:02.343 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:02.998 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.017642 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T18:40:02.998 INFO:tasks.rook.operator.smithi081.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T18:40:02.999 INFO:tasks.rook.operator.smithi081.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T18:40:02.999 INFO:tasks.rook.operator.smithi081.stdout: ... // 3 identical fields 2022-02-21T18:40:02.999 INFO:tasks.rook.operator.smithi081.stdout: Config: nil, 2022-02-21T18:40:02.999 INFO:tasks.rook.operator.smithi081.stdout: Selection: {}, 2022-02-21T18:40:02.999 INFO:tasks.rook.operator.smithi081.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T18:40:03.000 INFO:tasks.rook.operator.smithi081.stdout: ... // 2 identical elements 2022-02-21T18:40:03.000 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi081-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi081-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:03.000 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi081-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi081-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:03.000 INFO:tasks.rook.operator.smithi081.stdout:- { 2022-02-21T18:40:03.001 INFO:tasks.rook.operator.smithi081.stdout:- Name: "smithi140-lv-1", 2022-02-21T18:40:03.001 INFO:tasks.rook.operator.smithi081.stdout:- Count: 1, 2022-02-21T18:40:03.001 INFO:tasks.rook.operator.smithi081.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-02-21T18:40:03.001 INFO:tasks.rook.operator.smithi081.stdout:- }, 2022-02-21T18:40:03.002 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi140-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi140-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:03.002 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi140-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi140-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:03.002 INFO:tasks.rook.operator.smithi081.stdout: ... // 5 identical elements 2022-02-21T18:40:03.002 INFO:tasks.rook.operator.smithi081.stdout: }, 2022-02-21T18:40:03.003 INFO:tasks.rook.operator.smithi081.stdout: }, 2022-02-21T18:40:03.003 INFO:tasks.rook.operator.smithi081.stdout: Annotations: nil, 2022-02-21T18:40:03.003 INFO:tasks.rook.operator.smithi081.stdout: Labels: nil, 2022-02-21T18:40:03.003 INFO:tasks.rook.operator.smithi081.stdout: ... // 20 identical fields 2022-02-21T18:40:03.003 INFO:tasks.rook.operator.smithi081.stdout: } 2022-02-21T18:40:03.004 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.017813 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T18:40:03.004 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:40:03.018247 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T18:40:03.004 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.018287 I | operator: successfully started the controller-runtime manager 2022-02-21T18:40:03.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.067823 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T18:40:03.046 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.067849 I | operator: watching all namespaces for Ceph CRs 2022-02-21T18:40:03.046 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.067886 I | operator: setting up schemes 2022-02-21T18:40:03.049 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.071946 I | operator: setting up the controller-runtime manager 2022-02-21T18:40:03.353 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.375765 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T18:40:03.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392774 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T18:40:03.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392813 I | ceph-cluster-controller: successfully started 2022-02-21T18:40:03.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392867 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T18:40:03.373 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392891 I | ceph-crashcollector-controller: successfully started 2022-02-21T18:40:03.373 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392913 I | ceph-block-pool-controller: successfully started 2022-02-21T18:40:03.373 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392935 I | ceph-object-store-user-controller: successfully started 2022-02-21T18:40:03.373 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392953 I | ceph-object-realm-controller: successfully started 2022-02-21T18:40:03.374 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392978 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T18:40:03.374 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.392999 I | ceph-object-zone-controller: successfully started 2022-02-21T18:40:03.374 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393209 I | ceph-object-controller: successfully started 2022-02-21T18:40:03.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393281 I | ceph-file-controller: successfully started 2022-02-21T18:40:03.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393347 I | ceph-nfs-controller: successfully started 2022-02-21T18:40:03.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393397 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T18:40:03.376 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393448 I | ceph-client-controller: successfully started 2022-02-21T18:40:03.376 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393494 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T18:40:03.377 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393542 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T18:40:03.377 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393578 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T18:40:03.377 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393614 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T18:40:03.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393650 I | ceph-bucket-topic: successfully started 2022-02-21T18:40:03.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393679 I | ceph-bucket-notification: successfully started 2022-02-21T18:40:03.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393701 I | ceph-bucket-notification: successfully started 2022-02-21T18:40:03.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.393727 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T18:40:03.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.394810 I | operator: starting the controller-runtime manager 2022-02-21T18:40:03.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.497860 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T18:40:03.476 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.497905 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T18:40:03.476 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.497916 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T18:40:03.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.500865 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:40:03.497 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.519746 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T18:40:03.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.522664 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:03.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.522803 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:03.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.522882 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T18:40:03.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.522900 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:40:03.504 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.525217 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T18:40:03.504 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:40:03.525310 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:40:03.512 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.534091 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T18:40:03.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542720 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T18:40:03.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542742 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T18:40:03.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542749 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T18:40:03.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542755 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T18:40:03.524 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542763 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T18:40:03.524 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542771 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T18:40:03.524 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542780 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T18:40:03.524 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542790 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T18:40:03.525 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542798 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T18:40:03.525 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542806 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T18:40:03.525 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542814 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T18:40:03.525 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542823 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T18:40:03.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542833 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T18:40:03.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542840 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T18:40:03.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542848 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T18:40:03.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542856 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T18:40:03.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542941 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:03.528 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.542957 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:03.529 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:03.545953 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:40:06.726 INFO:teuthology.orchestra.run.smithi081.stdout:Success 2022-02-21T18:40:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:+ removed_pv= 2022-02-21T18:40:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' '' = '' ']' 2022-02-21T18:40:06.754 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl get pv 2022-02-21T18:40:06.754 INFO:teuthology.orchestra.run.smithi081.stderr:++ grep Released 2022-02-21T18:40:06.754 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f1 -d ' ' 2022-02-21T18:40:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:+ removed_pv= 2022-02-21T18:40:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 3s 2022-02-21T18:40:07.494 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:07.515920 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-02-21T18:40:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' '' = '' ']' 2022-02-21T18:40:09.816 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl get pv 2022-02-21T18:40:09.816 INFO:teuthology.orchestra.run.smithi081.stderr:++ grep Released 2022-02-21T18:40:09.816 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f1 -d ' ' 2022-02-21T18:40:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:+ removed_pv=smithi140-lv-1 2022-02-21T18:40:09.883 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 3s 2022-02-21T18:40:11.012 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.034754 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:40:11.013 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.034779 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:40:11.013 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.034789 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:40:11.073 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.094672 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:11.129 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.151858 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:11.129 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.152062 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:11.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277176 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T18:40:11.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277205 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T18:40:11.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277214 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:40:11.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277219 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:40:11.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277246 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:40:11.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277255 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:40:11.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277261 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:40:11.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277265 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:40:11.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277269 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:40:11.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277274 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:40:11.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277278 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:40:11.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277282 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:40:11.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277291 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:40:11.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277296 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:40:11.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277301 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T18:40:11.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277306 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T18:40:11.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277310 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T18:40:11.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277314 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T18:40:11.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277320 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T18:40:11.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277325 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:40:11.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277331 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:40:11.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277335 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T18:40:11.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277339 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T18:40:11.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277398 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T18:40:11.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.277404 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T18:40:11.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.286016 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T18:40:11.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292029 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:11.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292048 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292051 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:11.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292054 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292056 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:11.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292059 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.292062 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T18:40:11.405 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.428009 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:11.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.428026 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.428029 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:40:11.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.554247 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T18:40:11.539 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.561899 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:40:11.687 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.709312 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:11.687 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.709341 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.687 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.709355 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T18:40:11.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.750893 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:11.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.750925 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:11.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.750939 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:40:11.736 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.757559 I | op-mon: start running mons 2022-02-21T18:40:11.894 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.916558 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:11.952 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:11.974134 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T18:40:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' smithi140-lv-1 = '' ']' 2022-02-21T18:40:12.885 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl get pv smithi140-lv-1 -o 'jsonpath={.spec.local.path}' 2022-02-21T18:40:12.947 INFO:teuthology.orchestra.run.smithi081.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-02-21T18:40:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:++ echo smithi140-lv-1 2022-02-21T18:40:12.949 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f1 -d - 2022-02-21T18:40:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:+ host=smithi140 2022-02-21T18:40:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:+ toolbox ceph orch device zap smithi140 /dev/vg_nvme/lv_1 --force 2022-02-21T18:40:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi140 /dev/vg_nvme/lv_1 --force 2022-02-21T18:40:12.959 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:12.979147 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T18:40:12.961 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:12.984580 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T18:40:12.962 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:12.984612 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T18:40:12.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:12.989350 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T18:40:13.023 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:13.361 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:13.383536 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:40:13.540 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/vg_nvme/lv_1 on smithi140 zapped 2022-02-21T18:40:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:+ zap_completion=0 2022-02-21T18:40:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 0 = 0 ']' 2022-02-21T18:40:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-02-21T18:40:13.639 INFO:teuthology.orchestra.run.smithi081.stderr:+ zap_completion= 2022-02-21T18:40:13.639 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 3s 2022-02-21T18:40:13.959 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:13.978098 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:13.962 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:13.982125 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:15.558 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:15.580603 I | op-mon: targeting the mon count 3 2022-02-21T18:40:15.566 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:15.588833 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:16.006 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:16.028964 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:16.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:16.028987 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' '' = 0 ']' 2022-02-21T18:40:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:+ kubectl patch pv smithi140-lv-1 -p '{"spec":{"claimRef": null}}' 2022-02-21T18:40:16.752 INFO:teuthology.orchestra.run.smithi081.stdout:persistentvolume/smithi140-lv-1 patched 2022-02-21T18:40:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-02-21T18:40:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-02-21T18:40:16.812 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:16.853 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:16.874135 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:16.953 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:16.874157 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:17.317 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.339207 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:17.317 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.339237 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:17.498 INFO:teuthology.orchestra.run.smithi081.stdout:['Success'] 2022-02-21T18:40:17.508 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.524959 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T18:40:17.508 INFO:tasks.rook.operator.smithi081.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T18:40:17.508 INFO:tasks.rook.operator.smithi081.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T18:40:17.508 INFO:tasks.rook.operator.smithi081.stdout: ... // 3 identical fields 2022-02-21T18:40:17.509 INFO:tasks.rook.operator.smithi081.stdout: Config: nil, 2022-02-21T18:40:17.509 INFO:tasks.rook.operator.smithi081.stdout: Selection: {}, 2022-02-21T18:40:17.509 INFO:tasks.rook.operator.smithi081.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T18:40:17.509 INFO:tasks.rook.operator.smithi081.stdout: ... // 9 identical elements 2022-02-21T18:40:17.509 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi179-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi179-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:17.510 INFO:tasks.rook.operator.smithi081.stdout: {Name: "smithi179-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi179-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-02-21T18:40:17.510 INFO:tasks.rook.operator.smithi081.stdout:+ { 2022-02-21T18:40:17.510 INFO:tasks.rook.operator.smithi081.stdout:+ Name: "smithi140-lv-1", 2022-02-21T18:40:17.511 INFO:tasks.rook.operator.smithi081.stdout:+ Count: 1, 2022-02-21T18:40:17.511 INFO:tasks.rook.operator.smithi081.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-02-21T18:40:17.511 INFO:tasks.rook.operator.smithi081.stdout:+ }, 2022-02-21T18:40:17.511 INFO:tasks.rook.operator.smithi081.stdout: }, 2022-02-21T18:40:17.511 INFO:tasks.rook.operator.smithi081.stdout: }, 2022-02-21T18:40:17.512 INFO:tasks.rook.operator.smithi081.stdout: Annotations: nil, 2022-02-21T18:40:17.512 INFO:tasks.rook.operator.smithi081.stdout: Labels: nil, 2022-02-21T18:40:17.512 INFO:tasks.rook.operator.smithi081.stdout: ... // 20 identical fields 2022-02-21T18:40:17.512 INFO:tasks.rook.operator.smithi081.stdout: } 2022-02-21T18:40:17.512 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.525207 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T18:40:17.513 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:40:17.525855 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T18:40:17.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.533672 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T18:40:17.514 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.533688 I | operator: watching all namespaces for Ceph CRs 2022-02-21T18:40:17.514 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.533709 I | operator: setting up schemes 2022-02-21T18:40:17.526 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.535493 I | operator: setting up the controller-runtime manager 2022-02-21T18:40:17.533 INFO:teuthology.orchestra.run.smithi081.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-02-21T18:40:17.588 INFO:teuthology.orchestra.run.smithi081.stdout:job.batch "rook-ceph-device-zap" deleted 2022-02-21T18:40:17.600 INFO:teuthology.orchestra.run.smithi081.stderr:+ num_osd=0 2022-02-21T18:40:17.600 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 0 '!=' 12 ']' 2022-02-21T18:40:17.601 INFO:teuthology.orchestra.run.smithi081.stdout:waiting for osd to come back up 2022-02-21T18:40:17.602 INFO:teuthology.orchestra.run.smithi081.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T18:40:17.602 INFO:teuthology.orchestra.run.smithi081.stderr:++ toolbox ceph osd stat 2022-02-21T18:40:17.602 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T18:40:17.603 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f3 -d ' ' 2022-02-21T18:40:17.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.720676 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:17.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.720709 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:40:17.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.720762 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to remove legacy options: failed to delete one or more Ceph configs: failed to delete ceph config in the centralized mon configuration database. output: : context canceled 2022-02-21T18:40:17.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.720769 I | op-mon: checking for basic quorum with existing mons 2022-02-21T18:40:17.708 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:17.710 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.733398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-02-21T18:40:17.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.733519 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:40:17.721 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.744334 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:17.736 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.758751 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-02-21 18:40:17.750982661 +0000 UTC m=+608.480327888 LastTransitionTime:2022-02-21 18:40:17.75098258 +0000 UTC m=+608.480327810}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:40:17.736 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.758765 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:40:17.817 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.839957 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T18:40:17.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.848947 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T18:40:17.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.848979 I | ceph-cluster-controller: successfully started 2022-02-21T18:40:17.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849024 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T18:40:17.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849037 I | ceph-crashcollector-controller: successfully started 2022-02-21T18:40:17.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849052 I | ceph-block-pool-controller: successfully started 2022-02-21T18:40:17.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849065 I | ceph-object-store-user-controller: successfully started 2022-02-21T18:40:17.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849084 I | ceph-object-realm-controller: successfully started 2022-02-21T18:40:17.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849098 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T18:40:17.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849106 I | ceph-object-zone-controller: successfully started 2022-02-21T18:40:17.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849232 I | ceph-object-controller: successfully started 2022-02-21T18:40:17.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849263 I | ceph-file-controller: successfully started 2022-02-21T18:40:17.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849290 I | ceph-nfs-controller: successfully started 2022-02-21T18:40:17.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849313 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T18:40:17.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849334 I | ceph-client-controller: successfully started 2022-02-21T18:40:17.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849352 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T18:40:17.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849380 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T18:40:17.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849396 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T18:40:17.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849407 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T18:40:17.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849425 I | ceph-bucket-topic: successfully started 2022-02-21T18:40:17.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849441 I | ceph-bucket-notification: successfully started 2022-02-21T18:40:17.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849454 I | ceph-bucket-notification: successfully started 2022-02-21T18:40:17.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.849464 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T18:40:17.836 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.851012 I | operator: starting the controller-runtime manager 2022-02-21T18:40:17.933 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.955840 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T18:40:17.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.955889 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T18:40:17.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.955903 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T18:40:17.936 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.957424 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:40:17.942 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.965278 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T18:40:17.945 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.967195 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:17.945 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.967477 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:17.945 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.967522 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T18:40:17.946 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.967534 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:40:17.947 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.968796 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T18:40:17.947 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:40:17.968846 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T18:40:17.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.987973 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T18:40:17.968 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:17.991145 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:40:17.989 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010441 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T18:40:17.989 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010476 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T18:40:17.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010491 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T18:40:17.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010503 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T18:40:17.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010515 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T18:40:17.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010528 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T18:40:17.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010543 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T18:40:17.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010561 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T18:40:17.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010574 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T18:40:17.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010588 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T18:40:17.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010600 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T18:40:17.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010614 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T18:40:17.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010628 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T18:40:17.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010638 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T18:40:17.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010649 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T18:40:17.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010662 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T18:40:17.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010740 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:17.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.010764 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:18.158 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.178990 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-02-21T18:40:18.158 INFO:teuthology.orchestra.run.smithi081.stderr:+ num_osd=11 2022-02-21T18:40:18.159 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 30s 2022-02-21T18:40:18.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:18.246912 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-02-21T18:40:21.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:21.251022 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-02-21T18:40:24.234 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:24.256392 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-02-21T18:40:26.162 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183224 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T18:40:26.162 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183268 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T18:40:26.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183284 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:40:26.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183293 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T18:40:26.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183303 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:40:26.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183312 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T18:40:26.164 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183321 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:40:26.164 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183331 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T18:40:26.164 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183343 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:40:26.165 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183360 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T18:40:26.165 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183370 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:40:26.165 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183379 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T18:40:26.166 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183415 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:40:26.166 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183427 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T18:40:26.166 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183440 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T18:40:26.166 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183451 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T18:40:26.167 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183461 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T18:40:26.167 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183470 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T18:40:26.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183481 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T18:40:26.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183491 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:40:26.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183502 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T18:40:26.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183512 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T18:40:26.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183522 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T18:40:26.170 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183535 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T18:40:26.170 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.183546 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T18:40:26.170 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.190069 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T18:40:26.175 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198086 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:26.175 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198108 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198113 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:26.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198116 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198120 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:26.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198123 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.198127 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T18:40:26.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.313635 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:26.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.313667 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.313676 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:40:26.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.360983 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T18:40:26.555 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.577614 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T18:40:26.555 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.577634 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.577639 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T18:40:26.597 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.620001 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T18:40:26.598 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.620031 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T18:40:26.598 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.620043 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T18:40:26.640 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.662802 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T18:40:26.914 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.937098 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T18:40:26.919 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.941804 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T18:40:26.919 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.941832 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T18:40:26.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:26.945754 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T18:40:27.237 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:27.260543 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-02-21T18:40:34.299 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.321846 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:40:34.300 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.321868 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:40:34.300 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.321876 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:40:34.302 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.322171 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:40:34.302 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.322197 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T18:40:34.303 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.322214 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T18:40:34.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.406902 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:34.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.407029 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:34.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.438846 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:34.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.438981 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:34.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.439170 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:34.420 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.439552 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:34.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.893320 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:40:34.900 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:34.923009 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T18:40:35.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:35.043872 I | op-mon: start running mons 2022-02-21T18:40:35.025 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:35.044235 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-02-21 18:40:35.032206261 +0000 UTC m=+625.761551546 LastTransitionTime:2022-02-21 18:40:35.032206015 +0000 UTC m=+625.761551315}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:40:35.046 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:35.068196 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:36.024 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:36.047231 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:b=10.104.249.171:6789,c=10.105.216.166:6789,a=10.105.182.125:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:40:36.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:36.646037 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:36.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:36.646369 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:38.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.248641 I | op-mon: targeting the mon count 3 2022-02-21T18:40:38.234 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.257830 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:38.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.601924 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:38.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.601962 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:38.946 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.968489 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:38.946 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:38.968527 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:39.288 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:39.311025 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:39.289 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:39.311058 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:39.637 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:39.660036 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:39.638 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:39.660067 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:40:39.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:40.001004 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T18:40:39.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:40.001038 I | op-mon: checking for basic quorum with existing mons 2022-02-21T18:40:40.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:40.159660 I | op-mon: mon "c" endpoint is [v2:10.105.216.166:3300,v1:10.105.216.166:6789] 2022-02-21T18:40:40.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:40.269094 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:40:40.628 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:40.650649 I | op-mon: mon "b" endpoint is [v2:10.104.249.171:3300,v1:10.104.249.171:6789] 2022-02-21T18:40:41.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:41.249627 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:40:41.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:41.848362 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:41.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:41.848642 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:42.274 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:42.297314 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T18:40:42.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:42.305288 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T18:40:42.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:42.305313 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:42.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:42.858704 I | op-mon: mons running: [c a b] 2022-02-21T18:40:43.243 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.266047 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:43.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.275422 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T18:40:43.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.285324 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T18:40:43.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.285347 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:43.435 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.457705 I | op-mon: mons running: [c a b] 2022-02-21T18:40:43.850 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.873000 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:43.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.878932 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T18:40:43.865 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.888391 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T18:40:43.865 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:43.888409 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:44.048 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:44.070829 I | op-mon: mons running: [c a b] 2022-02-21T18:40:44.469 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:44.492162 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:44.470 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:44.492196 I | op-mon: mons created: 3 2022-02-21T18:40:44.881 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:44.904161 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:44.922 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:44.945521 I | op-mon: mons running: [c a b] 2022-02-21T18:40:45.343 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:45.366191 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:45.343 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:45.366225 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T18:40:45.344 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:45.366294 I | op-mon: start running mons 2022-02-21T18:40:45.348 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:45.371639 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:40:45.821 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:45.843729 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T18:40:46.025 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:46.047882 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.216.166:6789","10.105.182.125:6789","10.104.249.171:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:40:46.240 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:46.263192 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T18:40:46.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:46.649195 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:46.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:46.649340 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:46.648 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:46.671068 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T18:40:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T18:40:48.158 INFO:teuthology.orchestra.run.smithi081.stdout:waiting for osd to come back up 2022-02-21T18:40:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T18:40:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:++ toolbox ceph osd stat 2022-02-21T18:40:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T18:40:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f3 -d ' ' 2022-02-21T18:40:48.227 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:40:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:+ num_osd=11 2022-02-21T18:40:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 30s 2022-02-21T18:40:49.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.249496 I | op-mon: targeting the mon count 3 2022-02-21T18:40:49.234 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.257300 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:49.590 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.613343 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T18:40:49.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.613375 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:49.827 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.849492 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T18:40:49.827 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.849531 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T18:40:49.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.986804 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T18:40:49.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:49.986837 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:50.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.305568 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T18:40:50.309 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.332013 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T18:40:50.309 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.332044 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:50.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.671532 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T18:40:50.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.671564 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T18:40:50.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.720169 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T18:40:50.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.720202 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T18:40:50.698 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:50.720211 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T18:40:51.124 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.146848 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T18:40:51.126 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.149068 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T18:40:51.127 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.149094 I | op-mon: checking for basic quorum with existing mons 2022-02-21T18:40:51.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.246492 I | op-mgr: start running mgr 2022-02-21T18:40:51.224 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.246528 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T18:40:51.427 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:51.450368 I | op-mon: mon "c" endpoint is [v2:10.105.216.166:3300,v1:10.105.216.166:6789] 2022-02-21T18:40:52.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:52.677806 I | op-mon: mon "a" endpoint is [v2:10.105.182.125:3300,v1:10.105.182.125:6789] 2022-02-21T18:40:52.867 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:52.889398 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T18:40:52.876 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:52.899653 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T18:40:52.879 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:52.903014 I | op-mgr: setting services to point to mgr "a" 2022-02-21T18:40:54.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.249627 I | op-mon: mon "b" endpoint is [v2:10.104.249.171:3300,v1:10.104.249.171:6789] 2022-02-21T18:40:54.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.847813 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T18:40:54.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.847964 I | op-mgr: successful modules: balancer 2022-02-21T18:40:54.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.870035 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T18:40:54.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.870073 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T18:40:54.895 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:54.918174 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T18:40:55.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248578 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-1" 2022-02-21T18:40:55.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248609 I | op-osd: OSD PVC "smithi081-lv-1-data-0zzmvw" already exists 2022-02-21T18:40:55.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248621 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-2" 2022-02-21T18:40:55.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248631 I | op-osd: OSD PVC "smithi081-lv-2-data-08pkzc" already exists 2022-02-21T18:40:55.229 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248640 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-3" 2022-02-21T18:40:55.229 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248651 I | op-osd: OSD PVC "smithi081-lv-3-data-0tvdqv" already exists 2022-02-21T18:40:55.229 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248660 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-4" 2022-02-21T18:40:55.229 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248672 I | op-osd: OSD PVC "smithi081-lv-4-data-0tqs4h" already exists 2022-02-21T18:40:55.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248690 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-2" 2022-02-21T18:40:55.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248701 I | op-osd: OSD PVC "smithi140-lv-2-data-0s5fp2" already exists 2022-02-21T18:40:55.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248711 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-3" 2022-02-21T18:40:55.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248722 I | op-osd: OSD PVC "smithi140-lv-3-data-07nf5l" already exists 2022-02-21T18:40:55.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248730 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-4" 2022-02-21T18:40:55.231 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248741 I | op-osd: OSD PVC "smithi140-lv-4-data-05zd97" already exists 2022-02-21T18:40:55.231 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248748 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-1" 2022-02-21T18:40:55.231 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248759 I | op-osd: OSD PVC "smithi179-lv-1-data-0gqt5r" already exists 2022-02-21T18:40:55.231 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248768 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-2" 2022-02-21T18:40:55.232 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248782 I | op-osd: OSD PVC "smithi179-lv-2-data-0bk9j2" already exists 2022-02-21T18:40:55.232 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248794 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-3" 2022-02-21T18:40:55.232 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248807 I | op-osd: OSD PVC "smithi179-lv-3-data-0mvm5g" already exists 2022-02-21T18:40:55.232 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248816 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-4" 2022-02-21T18:40:55.233 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248826 I | op-osd: OSD PVC "smithi179-lv-4-data-0thzk9" already exists 2022-02-21T18:40:55.233 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.248837 I | op-osd: creating 1 new PVCs for device set "smithi140-lv-1" 2022-02-21T18:40:55.426 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.449453 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.249.171:6789","10.105.216.166:6789","10.105.182.125:6789"]}] data:c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 mapping:{"node":{"a":{"Name":"smithi179","Hostname":"smithi179","Address":"172.21.15.179"},"b":{"Name":"smithi140","Hostname":"smithi140","Address":"172.21.15.140"},"c":{"Name":"smithi081","Hostname":"smithi081","Address":"172.21.15.81"}}} maxMonId:2] 2022-02-21T18:40:55.629 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.651899 I | op-osd: successfully provisioned PVC "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:40:55.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696696 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:40:55.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696724 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-1-data-0zzmvw" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696733 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:40:55.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696739 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-2-data-08pkzc" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696746 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:40:55.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696752 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-3-data-0tvdqv" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696759 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:40:55.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696767 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-4-data-0tqs4h" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696775 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:40:55.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696782 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-2-data-0s5fp2" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696790 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:40:55.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696798 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-3-data-07nf5l" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696807 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:40:55.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696814 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-4-data-05zd97" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696822 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:40:55.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696829 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-1-data-0gqt5r" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696835 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:40:55.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696841 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-2-data-0bk9j2" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696848 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:40:55.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696855 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-3-data-0mvm5g" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696862 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:40:55.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696870 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-4-data-0thzk9" because OSD daemon using the PVC already exists 2022-02-21T18:40:55.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.696878 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:40:55.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.856513 I | op-mgr: successful modules: prometheus 2022-02-21T18:40:55.834 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.856690 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T18:40:55.834 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:55.856941 I | op-mgr: successful modules: dashboard 2022-02-21T18:40:56.448 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.471296 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:40:56.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.471326 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T18:40:56.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.471333 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T18:40:56.624 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.647065 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T18:40:56.624 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.647274 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T18:40:56.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.847864 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "starting" 2022-02-21T18:40:56.926 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.949697 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:40:56.931 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.954394 I | op-osd: updating OSD 1 on PVC "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:40:56.931 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:56.954417 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:40:57.099 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.121553 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:40:57.104 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.127068 I | op-osd: updating OSD 10 on PVC "smithi081-lv-2-data-08pkzc" 2022-02-21T18:40:57.104 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.127103 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:40:57.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.269822 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T18:40:57.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.275844 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T18:40:57.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.275861 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:57.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.294482 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:40:57.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.356297 I | op-osd: updating OSD 11 on PVC "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:40:57.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.356314 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:40:57.445 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.468514 I | op-mgr: successful modules: orchestrator modules 2022-02-21T18:40:57.842 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:57.865496 I | op-mon: mons running: [c a b] 2022-02-21T18:40:58.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.061424 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:40:58.144 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.163888 I | op-osd: updating OSD 2 on PVC "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:40:58.144 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.163920 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:40:58.302 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.324984 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:58.533 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.556491 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T18:40:58.942 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.965486 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T18:40:58.943 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:58.965526 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:40:58.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.001939 I | op-mon: mons running: [c a b] 2022-02-21T18:40:59.250 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.271041 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:40:59.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.398000 I | op-osd: updating OSD 3 on PVC "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:40:59.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.398033 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:40:59.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.554030 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:40:59.733 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:40:59.756195 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T18:41:00.140 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.163471 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T18:41:00.140 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.163507 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:41:00.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.212033 I | op-mon: mons running: [c a b] 2022-02-21T18:41:00.438 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.461781 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:00.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.558573 I | op-osd: updating OSD 4 on PVC "smithi140-lv-3-data-07nf5l" 2022-02-21T18:41:00.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.558595 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:41:00.950 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.973431 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:41:00.951 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:00.973468 I | op-mon: mons created: 3 2022-02-21T18:41:01.240 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:01.262744 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:01.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:01.357144 I | op-osd: updating OSD 5 on PVC "smithi179-lv-4-data-0thzk9" 2022-02-21T18:41:01.339 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:01.357177 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:41:01.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:01.895269 I | op-mon: waiting for mon quorum with [c a b] 2022-02-21T18:41:01.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:01.952217 I | op-mon: mons running: [c a b] 2022-02-21T18:41:02.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.061204 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:02.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.170816 I | op-osd: updating OSD 6 on PVC "smithi140-lv-4-data-05zd97" 2022-02-21T18:41:02.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.170852 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:41:02.314 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.337003 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T18:41:02.314 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.337051 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T18:41:02.839 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.862142 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:02.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.957527 I | op-osd: updating OSD 7 on PVC "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:41:02.935 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:02.957565 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:41:03.425 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:03.448172 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T18:41:03.641 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:03.663898 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:03.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:03.826836 I | op-osd: updating OSD 8 on PVC "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:41:03.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:03.826867 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:41:03.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:03.871380 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T18:41:04.278 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.300838 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T18:41:04.437 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.460754 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:04.541 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.564927 I | op-osd: updating OSD 9 on PVC "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:41:04.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.564969 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:41:04.938 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.961174 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T18:41:04.938 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:04.961206 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T18:41:05.458 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.481259 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T18:41:05.611 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.634539 I | op-osd: OSD orchestration status for node smithi140-lv-1-data-0jmpjv is "orchestrating" 2022-02-21T18:41:05.957 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.980103 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "orchestrating" 2022-02-21T18:41:05.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.993297 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T18:41:05.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.993340 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T18:41:05.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:05.993347 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T18:41:06.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.407434 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T18:41:06.420 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.442993 I | op-mgr: start running mgr 2022-02-21T18:41:06.420 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.443027 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T18:41:06.891 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.914370 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T18:41:06.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.925194 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T18:41:06.905 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:06.928433 I | op-mgr: setting services to point to mgr "a" 2022-02-21T18:41:07.093 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.116805 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T18:41:07.094 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.116869 I | op-mgr: successful modules: balancer 2022-02-21T18:41:07.104 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.127725 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T18:41:07.105 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.127764 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T18:41:07.178 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.201813 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T18:41:07.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206251 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-1" 2022-02-21T18:41:07.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206288 I | op-osd: OSD PVC "smithi081-lv-1-data-0zzmvw" already exists 2022-02-21T18:41:07.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206303 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-2" 2022-02-21T18:41:07.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206319 I | op-osd: OSD PVC "smithi081-lv-2-data-08pkzc" already exists 2022-02-21T18:41:07.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206330 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-3" 2022-02-21T18:41:07.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206350 I | op-osd: OSD PVC "smithi081-lv-3-data-0tvdqv" already exists 2022-02-21T18:41:07.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206363 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi081-lv-4" 2022-02-21T18:41:07.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206382 I | op-osd: OSD PVC "smithi081-lv-4-data-0tqs4h" already exists 2022-02-21T18:41:07.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206396 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-2" 2022-02-21T18:41:07.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206412 I | op-osd: OSD PVC "smithi140-lv-2-data-0s5fp2" already exists 2022-02-21T18:41:07.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206427 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-3" 2022-02-21T18:41:07.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206443 I | op-osd: OSD PVC "smithi140-lv-3-data-07nf5l" already exists 2022-02-21T18:41:07.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206453 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-4" 2022-02-21T18:41:07.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206468 I | op-osd: OSD PVC "smithi140-lv-4-data-05zd97" already exists 2022-02-21T18:41:07.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206483 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-1" 2022-02-21T18:41:07.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206498 I | op-osd: OSD PVC "smithi179-lv-1-data-0gqt5r" already exists 2022-02-21T18:41:07.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206514 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-2" 2022-02-21T18:41:07.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206530 I | op-osd: OSD PVC "smithi179-lv-2-data-0bk9j2" already exists 2022-02-21T18:41:07.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206544 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-3" 2022-02-21T18:41:07.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206560 I | op-osd: OSD PVC "smithi179-lv-3-data-0mvm5g" already exists 2022-02-21T18:41:07.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206572 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi179-lv-4" 2022-02-21T18:41:07.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206588 I | op-osd: OSD PVC "smithi179-lv-4-data-0thzk9" already exists 2022-02-21T18:41:07.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206600 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi140-lv-1" 2022-02-21T18:41:07.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.206616 I | op-osd: OSD PVC "smithi140-lv-1-data-0jmpjv" already exists 2022-02-21T18:41:07.255 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276232 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:41:07.255 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276260 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-1-data-0zzmvw" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276267 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:41:07.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276271 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-2-data-08pkzc" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276276 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:41:07.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276281 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-3-data-0tvdqv" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276286 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:41:07.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276291 I | op-osd: skipping OSD prepare job creation for PVC "smithi081-lv-4-data-0tqs4h" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276296 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:41:07.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276302 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-2-data-0s5fp2" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276308 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:41:07.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276312 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-3-data-07nf5l" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276317 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:41:07.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276323 I | op-osd: skipping OSD prepare job creation for PVC "smithi140-lv-4-data-05zd97" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276328 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:41:07.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276333 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-1-data-0gqt5r" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276339 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:41:07.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276343 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-2-data-0bk9j2" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276348 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:41:07.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276354 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-3-data-0mvm5g" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276359 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:41:07.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276364 I | op-osd: skipping OSD prepare job creation for PVC "smithi179-lv-4-data-0thzk9" because OSD daemon using the PVC already exists 2022-02-21T18:41:07.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.276369 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:07.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.526880 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "starting" 2022-02-21T18:41:07.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.541378 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi140-lv-1-data-0jmpjv. Status={Conditions:[] StartTime:2022-02-21 18:40:56 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-02-21T18:41:07.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.541400 I | op-osd: started OSD provisioning job for PVC "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:07.519 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.541411 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T18:41:07.519 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.541417 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T18:41:07.700 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.723924 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "starting" 2022-02-21T18:41:07.802 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.825608 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:07.807 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.831061 I | op-osd: updating OSD 1 on PVC "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:41:07.808 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:07.831093 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-2-data-0s5fp2" 2022-02-21T18:41:08.013 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.036106 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:08.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.044342 I | op-osd: updating OSD 10 on PVC "smithi081-lv-2-data-08pkzc" 2022-02-21T18:41:08.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.044370 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-2-data-08pkzc" 2022-02-21T18:41:08.180 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.203625 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:08.193 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.216924 I | op-osd: updating OSD 11 on PVC "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:41:08.193 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.216951 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-1-data-0zzmvw" 2022-02-21T18:41:08.639 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.661450 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:08.669 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.692288 I | op-mgr: successful modules: prometheus 2022-02-21T18:41:08.671 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.693498 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T18:41:08.672 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.693877 I | op-mgr: successful modules: dashboard 2022-02-21T18:41:08.734 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.757734 I | op-osd: updating OSD 2 on PVC "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:41:08.735 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:08.757768 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-2-data-0bk9j2" 2022-02-21T18:41:09.488 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:09.510046 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:09.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:09.558148 I | op-osd: updating OSD 3 on PVC "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:41:09.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:09.558180 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-1-data-0gqt5r" 2022-02-21T18:41:10.174 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:10.197442 I | op-mgr: successful modules: orchestrator modules 2022-02-21T18:41:10.242 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:10.265060 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:10.334 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:10.357622 I | op-osd: updating OSD 4 on PVC "smithi140-lv-3-data-07nf5l" 2022-02-21T18:41:10.335 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:10.357656 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-3-data-07nf5l" 2022-02-21T18:41:11.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.098966 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:11.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.157798 I | op-osd: updating OSD 5 on PVC "smithi179-lv-4-data-0thzk9" 2022-02-21T18:41:11.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.157830 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-4-data-0thzk9" 2022-02-21T18:41:11.841 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.863773 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:11.933 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.956294 I | op-osd: updating OSD 6 on PVC "smithi140-lv-4-data-05zd97" 2022-02-21T18:41:11.933 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:11.956320 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-4-data-05zd97" 2022-02-21T18:41:12.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.104775 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "completed" 2022-02-21T18:41:12.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.104818 I | op-osd: creating OSD 0 on PVC "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:12.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.104840 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:12.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.963138 I | op-osd: OSD orchestration status for PVC smithi140-lv-1-data-0jmpjv is "completed" 2022-02-21T18:41:12.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.963174 I | op-osd: creating OSD 0 on PVC "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:12.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:12.963192 I | op-osd: OSD will have its main bluestore block on "smithi140-lv-1-data-0jmpjv" 2022-02-21T18:41:13.365 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.385693 E | op-osd: failed to create OSD 0 on PVC "smithi140-lv-1-data-0jmpjv": failed to create deployment for OSD 0 on PVC "smithi140-lv-1-data-0jmpjv": deployments.apps "rook-ceph-osd-0" already exists 2022-02-21T18:41:13.465 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.488505 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:13.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.559987 I | op-osd: updating OSD 7 on PVC "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:41:13.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.560030 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-3-data-0tvdqv" 2022-02-21T18:41:13.718 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.741261 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T18:41:13.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:13.741293 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T18:41:14.240 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:14.263470 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:14.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:14.356029 I | op-osd: updating OSD 8 on PVC "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:41:14.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:14.356064 I | op-osd: OSD will have its main bluestore block on "smithi081-lv-4-data-0tqs4h" 2022-02-21T18:41:15.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:15.060868 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T18:41:15.153 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:15.176481 I | op-osd: updating OSD 9 on PVC "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:41:15.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:15.176505 I | op-osd: OSD will have its main bluestore block on "smithi179-lv-3-data-0mvm5g" 2022-02-21T18:41:15.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:41:15.809401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-02-21T18:41:15.786 INFO:tasks.rook.operator.smithi081.stdout:failed to create OSD 0 on PVC "smithi140-lv-1-data-0jmpjv": failed to create deployment for OSD 0 on PVC "smithi140-lv-1-data-0jmpjv": deployments.apps "rook-ceph-osd-0" already exists 2022-02-21T18:41:18.734 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T18:41:18.734 INFO:teuthology.orchestra.run.smithi081.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T18:41:18.735 INFO:teuthology.orchestra.run.smithi081.stdout:waiting for osd to come back up 2022-02-21T18:41:18.735 INFO:teuthology.orchestra.run.smithi081.stderr:++ toolbox ceph osd stat 2022-02-21T18:41:18.736 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T18:41:18.736 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f3 -d ' ' 2022-02-21T18:41:18.805 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:41:19.246 INFO:teuthology.orchestra.run.smithi081.stderr:+ num_osd=11 2022-02-21T18:41:19.246 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 30s 2022-02-21T18:41:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T18:41:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T18:41:49.248 INFO:teuthology.orchestra.run.smithi081.stdout:waiting for osd to come back up 2022-02-21T18:41:49.248 INFO:teuthology.orchestra.run.smithi081.stderr:++ toolbox ceph osd stat 2022-02-21T18:41:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T18:41:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:++ cut -f3 -d ' ' 2022-02-21T18:41:49.337 INFO:teuthology.orchestra.run.smithi081.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T18:41:49.814 INFO:teuthology.orchestra.run.smithi081.stderr:+ num_osd=12 2022-02-21T18:41:49.814 INFO:teuthology.orchestra.run.smithi081.stderr:+ sleep 30s 2022-02-21T18:42:19.815 INFO:teuthology.orchestra.run.smithi081.stderr:+ '[' 12 '!=' 12 ']' 2022-02-21T18:42:19.817 INFO:teuthology.run_tasks:Running task rook.shell... 2022-02-21T18:42:19.828 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch status 2022-02-21T18:42:23.129 INFO:teuthology.orchestra.run.smithi081.stdout:Backend: rook 2022-02-21T18:42:23.129 INFO:teuthology.orchestra.run.smithi081.stdout:Available: Yes 2022-02-21T18:42:23.161 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ps 2022-02-21T18:42:23.739 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-02-21T18:42:23.739 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.739 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi140 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T18:42:23.740 INFO:teuthology.orchestra.run.smithi081.stdout:crashcollector.crash smithi179 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T18:42:23.740 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.a smithi081 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T18:42:23.740 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a smithi179 running (10m) 0s ago 10m - - f61e120f25ae 2022-02-21T18:42:23.740 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b smithi140 running (10m) 0s ago 10m - - f61e120f25ae 2022-02-21T18:42:23.741 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c smithi081 running (7m) 0s ago 7m - - f61e120f25ae 2022-02-21T18:42:23.741 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi140 running (71s) 0s ago 71s - - f61e120f25ae 2022-02-21T18:42:23.741 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi140 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.741 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.742 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.743 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi179 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi179 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi140 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi179 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi140 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.745 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.745 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 smithi081 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.745 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 smithi179 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T18:42:23.760 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls 2022-02-21T18:42:24.409 INFO:teuthology.orchestra.run.smithi081.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-02-21T18:42:24.410 INFO:teuthology.orchestra.run.smithi081.stdout:crash 3/3 0s ago 5m * 2022-02-21T18:42:24.410 INFO:teuthology.orchestra.run.smithi081.stdout:mgr 1/1 0s ago 5m count:1 2022-02-21T18:42:24.410 INFO:teuthology.orchestra.run.smithi081.stdout:mon 3/3 0s ago 10m count:3 2022-02-21T18:42:24.411 INFO:teuthology.orchestra.run.smithi081.stdout:osd 24 0s ago 3m 2022-02-21T18:42:24.411 INFO:teuthology.orchestra.run.smithi081.stdout:osd.all-available-devices 0 0s ago - * 2022-02-21T18:42:24.436 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch host ls 2022-02-21T18:42:24.961 INFO:teuthology.orchestra.run.smithi081.stdout:HOST ADDR LABELS STATUS 2022-02-21T18:42:24.962 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 172.21.15.81/smithi081 2022-02-21T18:42:24.962 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 172.21.15.140/smithi140 2022-02-21T18:42:24.962 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 172.21.15.179/smithi179 2022-02-21T18:42:24.963 INFO:teuthology.orchestra.run.smithi081.stdout:3 hosts in cluster 2022-02-21T18:42:24.981 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch device ls 2022-02-21T18:42:25.564 INFO:teuthology.orchestra.run.smithi081.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2022-02-21T18:42:25.564 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:42:25.564 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:42:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:42:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:42:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:42:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:42:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:42:25.566 INFO:teuthology.orchestra.run.smithi081.stdout:smithi140 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:42:25.566 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T18:42:25.566 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T18:42:25.566 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T18:42:25.567 INFO:teuthology.orchestra.run.smithi081.stdout:smithi179 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T18:42:25.584 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch apply rgw foo 2022-02-21T18:42:26.149 INFO:teuthology.orchestra.run.smithi081.stdout:Created 2022-02-21T18:42:26.151 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.175171 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-02-21T18:42:26.171 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch apply mds foo 2022-02-21T18:42:26.274 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.298819 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:26.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.298890 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:42:26.835 INFO:teuthology.orchestra.run.smithi081.stdout:Created 2022-02-21T18:42:26.838 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.862204 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-02-21T18:42:26.853 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch apply rbd-mirror 2022-02-21T18:42:26.874 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.898416 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:42:26.879 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.903345 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:26.879 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:26.903442 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:42:27.557 INFO:teuthology.orchestra.run.smithi081.stdout:Success 2022-02-21T18:42:27.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:27.595675 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:27.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:27.595929 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:42:27.581 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch apply nfs foo --port 12777 2022-02-21T18:42:32.107 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:32.130942 I | ceph-spec: adding finalizer "cephnfs.ceph.rook.io" on "foo" 2022-02-21T18:42:32.108 INFO:teuthology.orchestra.run.smithi081.stdout:Created 2022-02-21T18:42:32.132 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-02-21T18:42:32.143 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:42:32.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:32.170013 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:32.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:32.170057 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T18:42:33.437 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:42:33.437 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:37:03.000000Z", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T18:42:32.000000Z", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:42:33.288842Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-02-21T18:42:33.288842Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:33.288842Z", "running": 0, "size": 1}}] 2022-02-21T18:42:33.456 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch rm mds.foo 2022-02-21T18:42:33.958 INFO:teuthology.orchestra.run.smithi081.stdout:Removed cephfilesystems/foo 2022-02-21T18:42:33.983 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch rm nfs.foo 2022-02-21T18:42:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:Removed cephnfses/foo 2022-02-21T18:42:34.651 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch rm rbd-mirror 2022-02-21T18:42:34.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:34.836627 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:42:35.211 INFO:teuthology.orchestra.run.smithi081.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-02-21T18:42:35.253 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch rm rgw.foo 2022-02-21T18:42:36.081 INFO:teuthology.orchestra.run.smithi081.stdout:Removed cephobjectstores/foo 2022-02-21T18:42:36.104 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:42:36.167 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:36.191599 I | ceph-object-controller: reconciling object store deployments 2022-02-21T18:42:36.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:36.282966 I | ceph-object-controller: ceph object store gateway service running at 10.101.165.123 2022-02-21T18:42:36.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:36.282992 I | ceph-object-controller: reconciling object store pools 2022-02-21T18:42:36.703 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:36.493893 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:42:36.703 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:36.556360 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:42:37.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:37.200328 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-02-21T18:42:37.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:42:37.240 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:37:03.000000Z", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T18:42:32.000000Z", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:42:37.104135Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:37.104135Z", "running": 0, "size": 1}}] 2022-02-21T18:42:37.337 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:37.358818 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-02-21T18:42:37.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:37.358918 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-02-21T18:42:37.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:37.358955 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-02-21T18:42:38.783 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:38.731115 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-02-21T18:42:38.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:38.868993 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-02-21T18:42:38.893 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:38.918071 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-02-21T18:42:39.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:39.184018 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi140": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:42:39.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:39.283537 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi140": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:42:39.965 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:39.989748 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-02-21T18:42:41.119 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:41.143683 I | op-mds: setting mds config flags 2022-02-21T18:42:41.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:41.143705 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T18:42:41.534 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:41.557606 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T18:42:41.963 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:41.987029 I | ceph-nfs-controller: configuring pool ".nfs" for nfs 2022-02-21T18:42:45.101 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:45.125094 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T18:42:45.139 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:45.163331 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-02-21T18:42:45.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:45.629515 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi179": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:42:46.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:46.396707 I | op-mds: setting mds config flags 2022-02-21T18:42:46.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:46.396743 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T18:42:47.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.045615 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-02-21T18:42:47.028 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.052830 I | ceph-nfs-controller: set pool ".nfs" for the application nfs 2022-02-21T18:42:47.031 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.053353 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-02-21T18:42:47.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.061756 I | ceph-nfs-controller: updating ceph nfs "foo" 2022-02-21T18:42:47.277 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:42:47.279 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.301432 I | cephclient: getting or creating ceph auth key "client.nfs-ganesha.foo.a" 2022-02-21T18:42:47.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.603007 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T18:42:47.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.833830 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-02-21T18:42:47.815 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.840114 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-02-21T18:42:47.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:47.865727 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-02-21T18:42:48.282 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:42:48.283 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:37:03.000000Z", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T18:42:32.000000Z", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:42:48.083435Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:48.083435Z", "running": 0, "size": 1}}] 2022-02-21T18:42:48.698 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:48.722952 I | ceph-nfs-controller: ceph nfs deployment "rook-ceph-nfs-foo-a" started 2022-02-21T18:42:48.768 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:48.792410 I | ceph-nfs-controller: ceph nfs service running at 10.106.100.135:2049 2022-02-21T18:42:48.768 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:48.792434 I | ceph-nfs-controller: adding ganesha "a" to grace db 2022-02-21T18:42:48.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:48.954400 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:48.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:48.954457 I | ceph-nfs-controller: deleting ceph nfs "foo" 2022-02-21T18:42:49.301 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:49.325399 I | ceph-file-controller: creating filesystem "foo" 2022-02-21T18:42:49.425 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:49.449185 I | ceph-nfs-controller: removing ganesha "a" from grace db 2022-02-21T18:42:49.497 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:49.521964 I | ceph-spec: removing finalizer "cephnfs.ceph.rook.io" on "foo" 2022-02-21T18:42:49.892 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:49.916186 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-02-21T18:42:50.297 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:50.321298 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:50.300 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:50.322575 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T18:42:50.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:50.513598 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T18:42:50.493 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:50.515293 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T18:42:55.101 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:55.125806 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-02-21T18:42:57.771 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:57.795549 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:42:57.783+0000 7fd50eb84340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:42:57.772 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:57.783+0000 7fd50eb84340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:42:57.772 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:57.783+0000 7fd50eb84340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:42:57.772 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:42:57.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:57.836914 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:58.306 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:42:58.391 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:58.415583 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:42:58.403+0000 7fa66b7c2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:42:58.392 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:58.403+0000 7fa66b7c2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:42:58.392 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:58.403+0000 7fa66b7c2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:42:58.392 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:42:58.454 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:58.477289 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:59.056 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:59.079970 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:42:59.067+0000 7f61fc6c1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:42:59.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:59.067+0000 7f61fc6c1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:42:59.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:59.067+0000 7f61fc6c1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:42:59.057 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:42:59.109 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:59.134150 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:42:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:42:59.151 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:37:03.000000Z", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:42:58.960563Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:42:58.960563Z", "running": 0, "size": 1}}] 2022-02-21T18:42:59.702 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:59.725864 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:42:59.715+0000 7f5178df1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:42:59.702 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:59.715+0000 7f5178df1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:42:59.702 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:42:59.715+0000 7f5178df1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:42:59.703 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:42:59.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:42:59.772933 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:00.778 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:00.802615 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:00.791+0000 7fc665869340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:00.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:00.791+0000 7fc665869340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:00.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:00.791+0000 7fc665869340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:00.779 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:00.861 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:00.885115 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-02-21T18:43:00.917 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:00.941595 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:01.482 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:01.506178 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:01.495+0000 7f41c75f2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:01.482 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:01.495+0000 7f41c75f2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:01.482 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:01.495+0000 7f41c75f2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:01.483 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:01.491 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:01.515711 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:01.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:01.616566 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-02-21T18:43:02.238 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:02.261359 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:02.251+0000 7fe5ad178340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:02.239 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:02.251+0000 7fe5ad178340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:02.239 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:02.251+0000 7fe5ad178340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:02.239 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:02.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:02.274336 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:02.836 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:02.860245 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:02.851+0000 7fcae032e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:02.837 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:02.851+0000 7fcae032e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:02.837 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:02.851+0000 7fcae032e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:02.837 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:02.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:02.907903 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:03.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:03.587900 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:03.579+0000 7f52be383340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:03.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:03.579+0000 7f52be383340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:03.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:03.579+0000 7f52be383340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:03.564 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:03.746 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:03.769974 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-02-21T18:43:03.747 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:03.769996 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-02-21T18:43:04.886 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:04.910387 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:04.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:04.936252 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-02-21T18:43:05.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:05.715764 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-02-21T18:43:06.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.157229 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:06.172 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.196436 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-02-21T18:43:06.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.196469 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-02-21T18:43:06.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.196477 I | cephclient: deleting ceph auth "mds.foo-a" 2022-02-21T18:43:06.765 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.788889 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:06.779+0000 7f5cb1960340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:06.765 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:06.779+0000 7f5cb1960340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:06.765 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:06.779+0000 7f5cb1960340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:06.765 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:06.859 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:06.883907 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-02-21T18:43:07.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:07.282341 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-02-21T18:43:07.776 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:07.800642 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-02-21T18:43:07.777 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:07.800679 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-02-21T18:43:07.777 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:07.800687 I | cephclient: deleting ceph auth "mds.foo-b" 2022-02-21T18:43:08.337 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:08.362065 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-02-21T18:43:08.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:08.362103 I | ceph-file-controller: downing filesystem "foo" 2022-02-21T18:43:09.178 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:43:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:43:09.911 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:43:09.741633Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:09.741633Z", "running": 0, "size": 1}}] 2022-02-21T18:43:09.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:10.020236 I | ceph-file-controller: downed filesystem "foo" 2022-02-21T18:43:11.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:11.940414 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:13.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:13.413408 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:13.403+0000 7fc966805340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:13.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:13.403+0000 7fc966805340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:13.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:13.403+0000 7fc966805340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:13.390 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:14.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:14.045555 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-02-21T18:43:14.021 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:14.045577 I | cephclient: purging pool "foo-metadata" (id=9) 2022-02-21T18:43:16.433 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:16.458282 I | cephclient: purge completed for pool "foo-metadata" 2022-02-21T18:43:16.815 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:16.840085 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-02-21T18:43:16.816 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:16.840111 I | cephclient: purging pool "foo-data0" (id=10) 2022-02-21T18:43:18.930 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:18.954599 I | cephclient: purge completed for pool "foo-data0" 2022-02-21T18:43:18.930 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:18.954702 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-02-21T18:43:18.956 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:18.981403 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-02-21T18:43:19.003 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:19.025789 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-02-21T18:43:19.010 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:19.026662 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-02-21T18:43:19.010 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:19.027180 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-02-21T18:43:19.937 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:43:21.028 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:43:21.028 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:43:20.860929Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:43:20.860929Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:43:20.860929Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:43:20.860929Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:43:20.860929Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:20.860929Z", "running": 0, "size": 1}}] 2022-02-21T18:43:21.451 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:21.476252 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi179": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:43:23.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:23.672032 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:24.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:24.272317 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:24.267+0000 7fb7c29aa340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:24.250 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:24.267+0000 7fb7c29aa340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:24.250 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:24.267+0000 7fb7c29aa340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:24.250 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:31.054 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:43:31.743 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:43:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:43:31.611459Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:43:31.611459Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:43:31.611459Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:43:31.611459Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:43:31.611459Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:31.611459Z", "running": 0, "size": 1}}] 2022-02-21T18:43:41.770 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:43:42.541 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:43:42.542 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:43:42.321460Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:43:42.321460Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:43:42.321460Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:43:42.321460Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:43:42.321460Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:42.321460Z", "running": 0, "size": 1}}] 2022-02-21T18:43:44.737 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:44.761696 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:43:45.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:43:45.407263 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:43:45.398+0000 7fdae6b34340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:43:45.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:45.398+0000 7fdae6b34340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:43:45.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:43:45.398+0000 7fdae6b34340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:43:45.383 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:43:52.561 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:43:53.280 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:43:53.280 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:43:53.137475Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:43:53.137475Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:43:53.137475Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:43:53.137475Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:43:53.137475Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:43:53.137475Z", "running": 0, "size": 1}}] 2022-02-21T18:44:03.306 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:04.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:04.020 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:03.872212Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:03.872212Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:03.872212Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:03.872212Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:03.872212Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:03.872212Z", "running": 0, "size": 1}}] 2022-02-21T18:44:14.045 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:14.715 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:14.585791Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:14.585791Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:14.585791Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:14.585791Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:14.585791Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:14.585791Z", "running": 0, "size": 1}}] 2022-02-21T18:44:24.743 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:25.506 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:25.507 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:25.336992Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:25.336992Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:25.336992Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:25.336992Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:25.336992Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:25.336992Z", "running": 0, "size": 1}}] 2022-02-21T18:44:26.348 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:44:26.374010 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:44:27.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:44:27.079429 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:44:27.070+0000 7f95f5f07340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:44:27.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:44:27.070+0000 7f95f5f07340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:44:27.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:44:27.070+0000 7f95f5f07340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:44:27.055 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:44:35.534 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:36.335 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:36.336 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:36.092693Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:36.092693Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:36.092693Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:36.092693Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:36.092693Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:36.092693Z", "running": 0, "size": 1}}] 2022-02-21T18:44:44.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:44:44.082700 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi140": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T18:44:46.365 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:47.197 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:47.197 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:46.888329Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:46.888329Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:46.888329Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:46.888329Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:46.888329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:46.888329Z", "running": 0, "size": 1}}] 2022-02-21T18:44:57.230 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:44:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:44:57.967 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:44:57.787172Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:44:57.787172Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:44:57.787172Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:44:57.787172Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:44:57.787172Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:44:57.787172Z", "running": 0, "size": 1}}] 2022-02-21T18:45:07.992 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:45:08.726 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:45:08.730 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:45:08.564694Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:45:08.564694Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:45:08.564694Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:45:08.564694Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:45:08.564694Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:45:08.564694Z", "running": 0, "size": 1}}] 2022-02-21T18:45:18.758 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:45:19.735 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:45:19.735 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:45:19.309603Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:45:19.309603Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:45:19.309603Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:45:19.309603Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:45:19.309603Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:45:19.309603Z", "running": 0, "size": 1}}] 2022-02-21T18:45:29.760 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:45:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:45:30.480 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:45:30.309913Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:45:30.309913Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:45:30.309913Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:45:30.309913Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:45:30.309913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:45:30.309913Z", "running": 0, "size": 1}}] 2022-02-21T18:45:40.504 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:45:41.215 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:45:41.216 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:45:41.070745Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:45:41.070745Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:45:41.070745Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:45:41.070745Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:45:41.070745Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:45:41.070745Z", "running": 0, "size": 1}}] 2022-02-21T18:45:48.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:45:49.017379 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:45:49.693 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:45:49.718391 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:45:49.704+0000 7fee0c116340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:45:49.693 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:45:49.704+0000 7fee0c116340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:45:49.694 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:45:49.704+0000 7fee0c116340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:45:49.694 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:45:51.270 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:45:52.001 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:45:52.001 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:45:51.860103Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:45:51.860103Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:45:51.860103Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:45:51.860103Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:45:51.860103Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:45:51.860103Z", "running": 0, "size": 1}}] 2022-02-21T18:46:02.029 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:02.725 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:02.726 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:02.587122Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:02.587122Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:02.587122Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:02.587122Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:02.587122Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:02.587122Z", "running": 0, "size": 1}}] 2022-02-21T18:46:12.753 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:13.438 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:13.438 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:13.301226Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:13.301226Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:13.301226Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:13.301226Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:13.301226Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:13.301226Z", "running": 0, "size": 1}}] 2022-02-21T18:46:23.466 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:24.174 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:24.174 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:24.012571Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:24.012571Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:24.012571Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:24.012571Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:24.012571Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:24.012571Z", "running": 0, "size": 1}}] 2022-02-21T18:46:34.200 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:35.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:35.255 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:34.786455Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:34.786455Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:34.786455Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:34.786455Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:34.786455Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:34.786455Z", "running": 0, "size": 1}}] 2022-02-21T18:46:45.315 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:45.988 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:45.988 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:45.846806Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:45.846806Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:45.846806Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:45.846806Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:45.846806Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:45.846806Z", "running": 0, "size": 1}}] 2022-02-21T18:46:56.016 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:46:56.764 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:46:56.765 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:46:56.577341Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:46:56.577341Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:46:56.577341Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:46:56.577341Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:46:56.577341Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:46:56.577341Z", "running": 0, "size": 1}}] 2022-02-21T18:47:06.796 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:47:07.520 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:47:07.552 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:47:07.382356Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:47:07.382356Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:47:07.382356Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:47:07.382356Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:47:07.382356Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:47:07.382356Z", "running": 0, "size": 1}}] 2022-02-21T18:47:17.555 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:47:18.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:47:18.240 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:47:18.099670Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:47:18.099670Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:47:18.099670Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:47:18.099670Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:47:18.099670Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:47:18.099670Z", "running": 0, "size": 1}}] 2022-02-21T18:47:28.265 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:47:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:47:29.007 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:47:28.797354Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:47:28.797354Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:47:28.797354Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:47:28.797354Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:47:28.797354Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:47:28.797354Z", "running": 0, "size": 1}}] 2022-02-21T18:47:39.033 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:47:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:47:39.724 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:47:39.590145Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:47:39.590145Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:47:39.590145Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:47:39.590145Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:47:39.590145Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:47:39.590145Z", "running": 0, "size": 1}}] 2022-02-21T18:47:49.750 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:47:50.528 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:47:50.528 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:47:50.334417Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:47:50.334417Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:47:50.334417Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:47:50.334417Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:47:50.334417Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:47:50.334417Z", "running": 0, "size": 1}}] 2022-02-21T18:48:00.553 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:01.324 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:01.325 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:01.098277Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:01.098277Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:01.098277Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:01.098277Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:01.098277Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:01.098277Z", "running": 0, "size": 1}}] 2022-02-21T18:48:11.349 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:12.020 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:12.020 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:11.880463Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:11.880463Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:11.880463Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:11.880463Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:11.880463Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:11.880463Z", "running": 0, "size": 1}}] 2022-02-21T18:48:22.046 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:22.739 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:22.739 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:22.606937Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:22.606937Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:22.606937Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:22.606937Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:22.606937Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:22.606937Z", "running": 0, "size": 1}}] 2022-02-21T18:48:32.763 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:33.489 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:33.489 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:33.317755Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:33.317755Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:33.317755Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:33.317755Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:33.317755Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:33.317755Z", "running": 0, "size": 1}}] 2022-02-21T18:48:33.538 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:48:33.565623 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:48:34.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:48:34.295504 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:48:34.285+0000 7fd76e5b3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:48:34.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:48:34.285+0000 7fd76e5b3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:48:34.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:48:34.285+0000 7fd76e5b3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:48:34.269 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:48:43.515 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:44.553 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:44.553 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:44.411637Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:44.411637Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:44.411637Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:44.411637Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:44.411637Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:44.411637Z", "running": 0, "size": 1}}] 2022-02-21T18:48:54.578 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:48:55.291 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:48:55.292 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:48:55.127628Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:48:55.127628Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:48:55.127628Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:48:55.127628Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:48:55.127628Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:48:55.127628Z", "running": 0, "size": 1}}] 2022-02-21T18:49:05.318 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:06.068 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:06.068 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:05.855396Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:05.855396Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:05.855396Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:05.855396Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:05.855396Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:05.855396Z", "running": 0, "size": 1}}] 2022-02-21T18:49:16.093 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:16.884 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:16.885 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:16.669151Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:16.669151Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:16.669151Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:16.669151Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:16.669151Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:16.669151Z", "running": 0, "size": 1}}] 2022-02-21T18:49:26.913 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:27.588 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:27.456485Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:27.456485Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:27.456485Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:27.456485Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:27.456485Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:27.456485Z", "running": 0, "size": 1}}] 2022-02-21T18:49:37.614 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:38.341 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:38.342 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:38.184315Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:38.184315Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:38.184315Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:38.184315Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:38.184315Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:38.184315Z", "running": 0, "size": 1}}] 2022-02-21T18:49:48.369 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:49.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:49.066 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:48.934093Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:48.934093Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:48.934093Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:48.934093Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:48.934093Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:48.934093Z", "running": 0, "size": 1}}] 2022-02-21T18:49:59.092 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:49:59.842 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:49:59.842 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:49:59.708683Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:49:59.708683Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:49:59.708683Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:49:59.708683Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:49:59.708683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:49:59.708683Z", "running": 0, "size": 1}}] 2022-02-21T18:50:09.869 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:50:10.561 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:50:10.562 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:50:10.411114Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:50:10.411114Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:50:10.411114Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:50:10.411114Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:50:10.411114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:50:10.411114Z", "running": 0, "size": 1}}] 2022-02-21T18:50:20.588 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:50:21.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:50:21.254 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:50:21.120996Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:50:21.120996Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:50:21.120996Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:50:21.120996Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:50:21.120996Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:50:21.120996Z", "running": 0, "size": 1}}] 2022-02-21T18:50:31.280 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:50:31.959 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:50:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:50:31.831265Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:50:31.831265Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:50:31.831265Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:50:31.831265Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:50:31.831265Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:50:31.831265Z", "running": 0, "size": 1}}] 2022-02-21T18:50:41.987 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:50:42.707 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:50:42.708 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:50:42.558002Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:50:42.558002Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:50:42.558002Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:50:42.558002Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:50:42.558002Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:50:42.558002Z", "running": 0, "size": 1}}] 2022-02-21T18:50:52.732 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:50:53.380 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:50:53.381 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:50:53.251519Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:50:53.251519Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:50:53.251519Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:50:53.251519Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:50:53.251519Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:50:53.251519Z", "running": 0, "size": 1}}] 2022-02-21T18:51:03.407 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:04.142 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:04.142 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:04.008117Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:04.008117Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:04.008117Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:04.008117Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:04.008117Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:04.008117Z", "running": 0, "size": 1}}] 2022-02-21T18:51:14.168 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:14.874 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:14.874 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:14.746039Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:14.746039Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:14.746039Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:14.746039Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:14.746039Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:14.746039Z", "running": 0, "size": 1}}] 2022-02-21T18:51:24.899 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:25.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:25.671 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:25.535339Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:25.535339Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:25.535339Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:25.535339Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:25.535339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:25.535339Z", "running": 0, "size": 1}}] 2022-02-21T18:51:35.695 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:36.468 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:36.302195Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:36.302195Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:36.302195Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:36.302195Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:36.302195Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:36.302195Z", "running": 0, "size": 1}}] 2022-02-21T18:51:46.495 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:47.217 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:47.217 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:47.062987Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:47.062987Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:47.062987Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:47.062987Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:47.062987Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:47.062987Z", "running": 0, "size": 1}}] 2022-02-21T18:51:57.243 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:51:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:51:57.975 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:51:57.838906Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:51:57.838906Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:51:57.838906Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:51:57.838906Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:51:57.838906Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:51:57.838906Z", "running": 0, "size": 1}}] 2022-02-21T18:52:08.000 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:52:08.699 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:52:08.699 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:52:08.565729Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:52:08.565729Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:52:08.565729Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:52:08.565729Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:52:08.565729Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:52:08.565729Z", "running": 0, "size": 1}}] 2022-02-21T18:52:18.726 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:52:19.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:52:19.441 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:52:19.277404Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:52:19.277404Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:52:19.277404Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:52:19.277404Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:52:19.277404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:52:19.277404Z", "running": 0, "size": 1}}] 2022-02-21T18:52:29.466 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:52:33.277 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:52:33.278 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:52:33.138370Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:52:33.138370Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:52:33.138370Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:52:33.138370Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:52:33.138370Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:52:33.138370Z", "running": 0, "size": 1}}] 2022-02-21T18:52:43.304 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:52:44.068 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:52:44.068 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:52:43.842468Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:52:43.842468Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:52:43.842468Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:52:43.842468Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:52:43.842468Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:52:43.842468Z", "running": 0, "size": 1}}] 2022-02-21T18:52:54.095 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:52:54.753 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:52:54.753 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:52:54.627830Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:52:54.627830Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:52:54.627830Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:52:54.627830Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:52:54.627830Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:52:54.627830Z", "running": 0, "size": 1}}] 2022-02-21T18:53:04.781 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:05.684 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:05.684 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:05.521698Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:05.521698Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:05.521698Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:05.521698Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:05.521698Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:05.521698Z", "running": 0, "size": 1}}] 2022-02-21T18:53:15.710 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:16.402 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:16.402 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:16.253126Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:16.253126Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:16.253126Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:16.253126Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:16.253126Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:16.253126Z", "running": 0, "size": 1}}] 2022-02-21T18:53:26.430 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:27.159 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:27.159 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:27.028269Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:27.028269Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:27.028269Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:27.028269Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:27.028269Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:27.028269Z", "running": 0, "size": 1}}] 2022-02-21T18:53:37.185 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:37.949 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:37.815500Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:37.815500Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:37.815500Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:37.815500Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:37.815500Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:37.815500Z", "running": 0, "size": 1}}] 2022-02-21T18:53:47.977 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:48.678 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:48.679 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:48.552373Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:48.552373Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:48.552373Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:48.552373Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:48.552373Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:48.552373Z", "running": 0, "size": 1}}] 2022-02-21T18:53:58.701 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:53:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:53:59.454 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:53:59.257916Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:53:59.257916Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:53:59.257916Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:53:59.257916Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:53:59.257916Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:53:59.257916Z", "running": 0, "size": 1}}] 2022-02-21T18:54:01.960 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:54:01.987373 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:54:02.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:54:02.684470 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:54:02.674+0000 7fad4ec54340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:54:02.657 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:54:02.674+0000 7fad4ec54340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:54:02.657 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:54:02.674+0000 7fad4ec54340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:54:02.657 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:54:09.477 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:54:10.299 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:54:10.299 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:54:10.038499Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:54:10.038499Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:54:10.038499Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:54:10.038499Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:54:10.038499Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:54:10.038499Z", "running": 0, "size": 1}}] 2022-02-21T18:54:20.326 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:54:21.050 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:54:21.050 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:54:20.915517Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:54:20.915517Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:54:20.915517Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:54:20.915517Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:54:20.915517Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:54:20.915517Z", "running": 0, "size": 1}}] 2022-02-21T18:54:31.075 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:54:31.840 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:54:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:54:31.710781Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:54:31.710781Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:54:31.710781Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:54:31.710781Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:54:31.710781Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:54:31.710781Z", "running": 0, "size": 1}}] 2022-02-21T18:54:41.871 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:54:42.593 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:54:43.124 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:54:42.449585Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:54:42.449585Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:54:42.449585Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:54:42.449585Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:54:42.449585Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:54:42.449585Z", "running": 0, "size": 1}}] 2022-02-21T18:54:53.127 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:54:53.845 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:54:53.846 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:54:53.711708Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:54:53.711708Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:54:53.711708Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:54:53.711708Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:54:53.711708Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:54:53.711708Z", "running": 0, "size": 1}}] 2022-02-21T18:55:03.874 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:04.569 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:04.569 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:04.437207Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:04.437207Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:04.437207Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:04.437207Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:04.437207Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:04.437207Z", "running": 0, "size": 1}}] 2022-02-21T18:55:14.598 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:15.352 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:15.353 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:15.170544Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:15.170544Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:15.170544Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:15.170544Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:15.170544Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:15.170544Z", "running": 0, "size": 1}}] 2022-02-21T18:55:25.376 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:26.137 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:26.137 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:25.974950Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:25.974950Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:25.974950Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:25.974950Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:25.974950Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:25.974950Z", "running": 0, "size": 1}}] 2022-02-21T18:55:36.169 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:36.885 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:36.885 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:36.758700Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:36.758700Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:36.758700Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:36.758700Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:36.758700Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:36.758700Z", "running": 0, "size": 1}}] 2022-02-21T18:55:46.918 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:47.643 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:47.643 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:47.484485Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:47.484485Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:47.484485Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:47.484485Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:47.484485Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:47.484485Z", "running": 0, "size": 1}}] 2022-02-21T18:55:57.669 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:55:58.351 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:55:58.351 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:55:58.216644Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:55:58.216644Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:55:58.216644Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:55:58.216644Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:55:58.216644Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:55:58.216644Z", "running": 0, "size": 1}}] 2022-02-21T18:56:08.377 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:56:09.075 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:56:09.076 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:56:08.934304Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:56:08.934304Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:56:08.934304Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:56:08.934304Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:56:08.934304Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:56:08.934304Z", "running": 0, "size": 1}}] 2022-02-21T18:56:19.103 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:56:19.824 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:56:19.825 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:56:19.679800Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:56:19.679800Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:56:19.679800Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:56:19.679800Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:56:19.679800Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:56:19.679800Z", "running": 0, "size": 1}}] 2022-02-21T18:56:29.851 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:56:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:56:30.588 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:56:30.387285Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:56:30.387285Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:56:30.387285Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:56:30.387285Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:56:30.387285Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:56:30.387285Z", "running": 0, "size": 1}}] 2022-02-21T18:56:40.615 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:56:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:56:41.365 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:56:41.203982Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:56:41.203982Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:56:41.203982Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:56:41.203982Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:56:41.203982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:56:41.203982Z", "running": 0, "size": 1}}] 2022-02-21T18:56:51.391 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:56:52.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:56:52.186 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:56:51.925493Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:56:51.925493Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:56:51.925493Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:56:51.925493Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:56:51.925493Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:56:51.925493Z", "running": 0, "size": 1}}] 2022-02-21T18:57:02.211 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:03.101 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:03.101 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:02.944851Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:02.944851Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:02.944851Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:02.944851Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:02.944851Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:02.944851Z", "running": 0, "size": 1}}] 2022-02-21T18:57:13.128 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:13.839 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:13.839 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:13.680153Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:13.680153Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:13.680153Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:13.680153Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:13.680153Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:13.680153Z", "running": 0, "size": 1}}] 2022-02-21T18:57:23.865 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:24.549 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:24.550 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:24.414981Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:24.414981Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:24.414981Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:24.414981Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:24.414981Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:24.414981Z", "running": 0, "size": 1}}] 2022-02-21T18:57:34.576 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:35.258 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:35.258 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:35.134180Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:35.134180Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:35.134180Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:35.134180Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:35.134180Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:35.134180Z", "running": 0, "size": 1}}] 2022-02-21T18:57:45.300 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:46.003 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:46.003 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:45.862841Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:45.862841Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:45.862841Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:45.862841Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:45.862841Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:45.862841Z", "running": 0, "size": 1}}] 2022-02-21T18:57:56.029 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:57:56.716 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:57:56.716 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:57:56.583267Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:57:56.583267Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:57:56.583267Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:57:56.583267Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:57:56.583267Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:57:56.583267Z", "running": 0, "size": 1}}] 2022-02-21T18:58:06.739 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:58:07.440 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:58:07.440 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:58:07.300696Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:58:07.300696Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:58:07.300696Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:58:07.300696Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:58:07.300696Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:58:07.300696Z", "running": 0, "size": 1}}] 2022-02-21T18:58:17.465 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:58:18.207 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:58:18.208 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:58:17.986026Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:58:17.986026Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:58:17.986026Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:58:17.986026Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:58:17.986026Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:58:17.986026Z", "running": 0, "size": 1}}] 2022-02-21T18:58:28.228 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:58:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:58:28.947 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:58:28.793247Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:58:28.793247Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:58:28.793247Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:58:28.793247Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:58:28.793247Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:58:28.793247Z", "running": 0, "size": 1}}] 2022-02-21T18:58:38.978 DEBUG:teuthology.orchestra.run.smithi081:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-nnkrk -- ceph orch ls -f json 2022-02-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-02-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi081.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:43.000000Z", "last_refresh": "2022-02-21T18:58:39.562468Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:36:56.000000Z", "last_refresh": "2022-02-21T18:58:39.562468Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:31:33.000000Z", "last_refresh": "2022-02-21T18:58:39.562468Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T18:38:38.000000Z", "last_refresh": "2022-02-21T18:58:39.562468Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T18:58:39.562468Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T18:58:39.562468Z", "running": 0, "size": 1}}] 2022-02-21T18:58:39.717 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T18:58:39.718 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T18:58:39.719 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-02-21T18:58:39.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T18:58:39.758 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T18:58:39.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T18:58:39.821 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T18:58:39.822 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T18:58:39.847 DEBUG:teuthology.orchestra.remote:smithi081:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T18:58:39.884 DEBUG:teuthology.orchestra.run.smithi081:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-02-21T18:58:39.954 INFO:teuthology.orchestra.run.smithi081.stdout:deployment.apps "rook-ceph-tools" deleted 2022-02-21T18:58:39.977 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T18:58:39.978 DEBUG:teuthology.orchestra.run.smithi081:> kubectl delete -f cluster.yaml 2022-02-21T18:58:40.057 INFO:teuthology.orchestra.run.smithi081.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-02-21T18:58:40.061 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.092434 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-02-21T18:58:40.065 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.092552 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T18:58:40.065 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.092981 I | operator: successfully started the controller-runtime manager 2022-02-21T18:58:40.065 INFO:tasks.rook.operator.smithi081.stdout:I0221 18:58:40.093005 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T18:58:40.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.111530 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T18:58:40.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.111554 I | operator: watching all namespaces for Ceph CRs 2022-02-21T18:58:40.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.111595 I | operator: setting up schemes 2022-02-21T18:58:40.085 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.115348 I | operator: setting up the controller-runtime manager 2022-02-21T18:58:40.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.421319 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T18:58:40.394 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424368 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T18:58:40.395 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424406 I | ceph-cluster-controller: successfully started 2022-02-21T18:58:40.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424456 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T18:58:40.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424477 I | ceph-crashcollector-controller: successfully started 2022-02-21T18:58:40.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424497 I | ceph-block-pool-controller: successfully started 2022-02-21T18:58:40.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424515 I | ceph-object-store-user-controller: successfully started 2022-02-21T18:58:40.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424539 I | ceph-object-realm-controller: successfully started 2022-02-21T18:58:40.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424550 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T18:58:40.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424561 I | ceph-object-zone-controller: successfully started 2022-02-21T18:58:40.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424683 I | ceph-object-controller: successfully started 2022-02-21T18:58:40.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424719 I | ceph-file-controller: successfully started 2022-02-21T18:58:40.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424748 I | ceph-nfs-controller: successfully started 2022-02-21T18:58:40.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424773 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T18:58:40.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424795 I | ceph-client-controller: successfully started 2022-02-21T18:58:40.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424815 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T18:58:40.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424842 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T18:58:40.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424861 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T18:58:40.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424877 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T18:58:40.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424891 I | ceph-bucket-topic: successfully started 2022-02-21T18:58:40.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424918 I | ceph-bucket-notification: successfully started 2022-02-21T18:58:40.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424942 I | ceph-bucket-notification: successfully started 2022-02-21T18:58:40.402 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.424960 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T18:58:40.402 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.426160 I | operator: starting the controller-runtime manager 2022-02-21T18:58:40.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.532009 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T18:58:40.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.532074 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T18:58:40.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.532095 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T18:58:40.506 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.535472 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T18:58:40.506 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.535524 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:40.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.656807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:40.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:40.666886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:41.141 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:41.172007 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:41.162+0000 7f3875891340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:41.142 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:41.162+0000 7f3875891340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:41.142 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:41.162+0000 7f3875891340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:41.142 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:41.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:41.190833 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:41.798 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:41.829271 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:41.818+0000 7f8e958f5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:41.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:41.818+0000 7f8e958f5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:41.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:41.818+0000 7f8e958f5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:41.799 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:41.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:41.895625 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:42.512 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:42.543097 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:42.534+0000 7f0e04436340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:42.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:42.534+0000 7f0e04436340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:42.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:42.534+0000 7f0e04436340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:42.513 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:42.538 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:42.569345 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:43.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:43.213694 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:43.206+0000 7f79cb5e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:43.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:43.206+0000 7f79cb5e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:43.184 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:43.206+0000 7f79cb5e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:43.184 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:43.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:43.284066 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:44.116 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:44.146966 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:44.138+0000 7f60a0514340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:44.117 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:44.138+0000 7f60a0514340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:44.117 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:44.138+0000 7f60a0514340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:44.117 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:44.218 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:44.249203 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:44.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:44.970537 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:44.962+0000 7fbe6ecce340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:44.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:44.962+0000 7fbe6ecce340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:44.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:44.962+0000 7fbe6ecce340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:44.942 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:45.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:45.138506 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:45.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:45.809523 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:45.798+0000 7fee9400d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:45.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:45.798+0000 7fee9400d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:45.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:45.798+0000 7fee9400d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:45.781 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:46.106 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:46.137394 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:46.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:46.826554 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:46.818+0000 7fb59c262340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:46.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:46.818+0000 7fb59c262340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:46.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:46.818+0000 7fb59c262340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:46.797 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:47.452 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:47.475047 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:48.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:48.177951 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:48.170+0000 7f622a20e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:48.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:48.170+0000 7f622a20e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:48.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:48.170+0000 7f622a20e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:48.148 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:49.435 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:49.466454 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:50.171 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:50.201602 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:50.194+0000 7f01726be340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:50.171 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:50.194+0000 7f01726be340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:50.171 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:50.194+0000 7f01726be340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:50.172 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:50.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:50.818734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:50.800 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:50.831520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:52.741 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:52.771972 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:53.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:53.497010 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:53.490+0000 7fdf4fb1e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:53.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:53.490+0000 7fdf4fb1e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:53.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:53.490+0000 7fdf4fb1e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:53.467 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:58:58.594 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:58.624660 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:58:59.224 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:58:59.254896 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:58:59.246+0000 7fbc829f0340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:58:59.224 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:59.246+0000 7fbc829f0340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:58:59.225 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:58:59.246+0000 7fbc829f0340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:58:59.225 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:59:00.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:00.958094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:00.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:00.972616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:09.471 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:09.501907 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:59:10.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:10.226587 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:59:10.217+0000 7f057fe21340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:59:10.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:59:10.217+0000 7f057fe21340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:59:10.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:59:10.217+0000 7f057fe21340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:59:10.197 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:59:11.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:11.233779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:11.216 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:11.247358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:21.321 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:21.352392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:21.335 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:21.366560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:30.684 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:30.715086 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T18:59:31.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:31.419761 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:59:31.409+0000 7f5420fa0340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:59:31.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:59:31.409+0000 7f5420fa0340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:59:31.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T18:59:31.409+0000 7f5420fa0340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:59:31.390 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:59:31.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:31.498342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:31.480 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:31.511308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:41.601 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:41.631861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:41.613 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:41.644822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:51.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:51.704649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:51.684 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 18:59:51.716133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:01.803 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:01.833833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:01.815 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:01.846872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:11.882 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:11.913168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:11.894 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:11.925118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:12.355 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:12.386170 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:00:13.009 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:13.040157 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:00:13.028+0000 7f44c3ea5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:00:13.010 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:00:13.028+0000 7f44c3ea5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:00:13.011 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:00:13.028+0000 7f44c3ea5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:00:13.011 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:00:22.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:22.073111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:22.058 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:22.088009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:32.132 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:32.163253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:32.143 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:32.174864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:42.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:42.280829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:42.265 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:42.296138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:52.327 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:52.357585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:52.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:00:52.369640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:02.500 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:02.531925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:02.513 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:02.544472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:12.578 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:12.609930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:12.627 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:12.658823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:22.801 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:22.833019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:22.816 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:22.847423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:32.894 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:32.925965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:32.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:32.937353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:34.939 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:34.970656 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:01:35.627 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:35.658834 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:01:35.650+0000 7f84b5883340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:01:35.628 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:01:35.650+0000 7f84b5883340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:01:35.628 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:01:35.650+0000 7f84b5883340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:01:35.628 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:01:43.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:43.046939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:43.053 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:43.084297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:53.128 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:53.159533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:53.141 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:01:53.173551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:03.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:03.255335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:03.235 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:03.267728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:13.350 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:13.381716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:13.362 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:13.394581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:23.495 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:23.527453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:23.508 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:23.540623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:33.612 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:33.643802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:33.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:33.655670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:43.708 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:43.740081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:43.746 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:43.778616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:53.877 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:53.909146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:53.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:02:53.922249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:03.963 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:03.995089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:03.976 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:04.008306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:14.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:14.089175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:14.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:14.103601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:24.209 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:24.241788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:24.222 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:24.254897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:34.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:34.323292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:34.385 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:34.418282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:44.555 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:44.587432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:44.566 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:44.598653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:54.654 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:54.686852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:54.669 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:03:54.701545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:04.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:04.761229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:04.742 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:04.774689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:14.876 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:14.909026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:14.896 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:14.928616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:19.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:19.505295 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:04:20.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:20.180099 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:04:20.170+0000 7f9aa3ca2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:04:20.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:04:20.170+0000 7f9aa3ca2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:04:20.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:04:20.170+0000 7f9aa3ca2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:04:20.149 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:04:25.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:25.070479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:25.047 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:25.079972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:35.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:35.152429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:35.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:35.165940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:45.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:45.295994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:45.274 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:45.307546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:55.387 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:55.420096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:55.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:04:55.433090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:05.474 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:05.507766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:05.488 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:05.521672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:15.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:15.624073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:15.602 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:15.635854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:25.717 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:25.750884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:25.728 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:25.762248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:35.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:35.828477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:35.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:35.888614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:45.920 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:45.953260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:45.930 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:45.964301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:56.095 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:56.128141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:56.111 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:05:56.144988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:06.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:06.222246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:06.201 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:06.235451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:16.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:16.305217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:16.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:16.316486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:26.448 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:26.482027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:26.459 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:26.493380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:36.573 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:36.607463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:36.587 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:36.621123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:46.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:46.839360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:46.819 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:46.853676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:56.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:56.974315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:56.953 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:06:56.987602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:07.128 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:07.162474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:07.141 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:07.175321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:17.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:17.416886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:17.394 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:17.428692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:27.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:27.556508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:27.554 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:27.588710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:38.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:38.216920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:38.197 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:38.232038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:48.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:48.451305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:48.429 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:48.463648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:58.496 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:58.530231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:58.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:07:58.597818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:08.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:08.879545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:08.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:08.889749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:19.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:19.286963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:19.264 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:19.299451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:29.408 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:29.442729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:29.422 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:29.456973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:39.571 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:39.606552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:39.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:39.619667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:49.817 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:49.851873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:49.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:49.866975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:59.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:08:59.999607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:59.975 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:00.010585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:10.326 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:10.360829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:10.340 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:10.375235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:20.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:20.521984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:20.499 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:20.534835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:30.609 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:30.645119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:30.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:30.659145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:40.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:40.732431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:41.014 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:40.851435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:47.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:47.867080 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:09:48.603 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:48.637715 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:09:48.624+0000 7f9f75f4e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:09:48.603 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:09:48.624+0000 7f9f75f4e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:09:48.603 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:09:48.624+0000 7f9f75f4e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:09:48.603 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:09:50.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:50.948090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:50.925 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:09:50.961462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:01.060 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:01.095317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:01.073 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:01.109065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:11.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:11.199370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:11.175 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:11.211294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:21.309 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:21.345122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:21.322 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:21.358638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:31.444 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:31.480033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:31.461 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:31.497446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:41.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:41.571119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:41.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:41.584009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:51.643 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:51.679076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:51.658 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:10:51.694611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:01.727 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:01.763656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:01.739 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:01.775338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:11.850 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:11.887063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:11.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:11.898903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:21.930 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:21.967010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:21.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:22.014904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:32.054 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:32.090293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:32.066 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:32.103263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:42.143 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:42.179220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:42.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:42.190747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:52.239 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:52.275903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:52.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:11:52.307575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:02.387 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:02.423898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:02.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:02.437886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:12.510 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:12.546921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:12.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:12.560841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:22.596 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:22.633076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:22.609 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:22.646123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:32.674 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:32.711632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:32.683 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:32.721054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:42.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:42.885265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:42.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:42.900010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:52.958 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:52.995577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:52.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:12:53.007898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:03.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:03.107820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:03.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:03.120764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:13.194 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:13.231679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:13.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:13.243428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:23.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:23.319090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:23.330 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:23.367249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:33.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:33.527258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:33.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:33.538712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:43.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:43.623756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:43.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:43.692465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:53.722 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:53.759330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:53.733 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:13:53.771134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:03.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:03.833509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:03.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:03.843000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:13.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:13.939742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:13.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:13.953903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:24.032 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:24.069807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:24.047 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:24.084567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:34.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:34.145914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:34.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:34.214314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:44.279 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:44.316855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:44.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:44.329923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:54.367 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:54.404464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:54.412 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:14:54.450317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:04.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:04.524184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:04.500 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:04.538244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:14.567 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:14.605491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:14.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:14.684799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:24.731 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:24.769111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:24.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:24.782056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:34.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:34.870835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:34.846 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:34.884416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:44.931 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:44.969625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:44.944 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:44.982659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:55.044 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:55.081992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:55.102 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:15:55.140781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:05.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:05.370499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:05.359 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:05.397067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:15.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:15.491223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:15.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:15.555887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:25.602 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:25.640610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:25.615 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:25.653070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:35.687 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:35.725450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:35.701 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:35.739932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:45.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:45.842551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:45.816 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:45.854738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:55.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:55.928292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:55.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:16:55.941509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:05.968 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:06.006277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:05.983 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:06.021316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:16.051 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:16.089839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:16.117 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:16.155838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:26.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:26.236357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:26.212 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:26.250799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:36.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:36.401293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:36.401 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:36.439665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:46.496 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:46.534087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:46.509 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:46.547514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:56.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:56.688897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:56.662 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:17:56.700593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:06.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:06.824951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:06.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:06.834775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:16.873 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:16.911791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:16.885 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:16.924050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:26.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:27.030692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:27.006 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:27.044853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:37.102 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:37.140766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:37.115 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:37.154578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:47.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:47.268793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:47.255 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:47.293672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:57.347 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:57.386205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:57.359 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:18:57.398002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:07.440 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:07.479204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:07.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:07.491956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:17.533 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:17.571752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:17.544 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:17.583925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:27.601 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:27.640563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:27.615 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:27.653940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:37.688 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:37.727635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:37.701 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:37.740794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:47.767 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:47.806630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:47.781 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:47.820170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:57.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:57.950842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:57.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:19:57.966219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:08.025 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:08.063871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:08.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:08.076917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:18.129 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:18.168200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:18.143 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:18.182108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:28.244 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:28.282917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:28.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:28.295225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:38.358 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:38.396898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:38.367 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:38.406492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:43.973 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:44.012490 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:20:44.706 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:44.744686 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:20:44.732+0000 7f2c50e53340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:20:44.706 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:20:44.732+0000 7f2c50e53340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:20:44.707 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:20:44.732+0000 7f2c50e53340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:20:44.707 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:20:48.433 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:48.472520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:48.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:48.489058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:58.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:58.556470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:58.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:20:58.569928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:08.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:08.674451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:08.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:08.692230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:18.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:18.835552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:18.807 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:18.846678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:28.915 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:28.953818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:28.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:28.968117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:39.029 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:39.068090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:39.041 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:39.080636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:49.102 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:49.141371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:49.116 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:49.155464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:59.195 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:59.233963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:59.207 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:21:59.246462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:09.280 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:09.319373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:09.294 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:09.333081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:19.360 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:19.398968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:19.496 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:19.416449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:29.482 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:29.521227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:29.494 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:29.533646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:39.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:39.628246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:39.600 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:39.639634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:49.667 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:49.706470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:49.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:49.716576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:59.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:59.967939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:59.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:22:59.980218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:10.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:10.208762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:10.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:10.221895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:20.425 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:20.463973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:20.439 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:20.478362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:30.615 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:30.654953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:30.630 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:30.669469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:40.750 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:40.789159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:40.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:40.803100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:50.823 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:50.862507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:50.873 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:23:50.913019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:00.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:01.033135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:01.008 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:01.048136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:11.089 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:11.128811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:11.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:11.203202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:21.300 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:21.340001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:21.314 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:21.353550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:31.463 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:31.503102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:31.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:31.514300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:41.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:41.686707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:41.664 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:41.703382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:51.715 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:51.754261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:51.728 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:24:51.767772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:01.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:01.867958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:01.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:01.884853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:11.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:12.003548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:11.977 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:12.016424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:22.095 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:22.134529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:22.109 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:22.148793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:32.204 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:32.243795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:32.219 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:32.259388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:42.322 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:42.361621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:42.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:42.373075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:52.456 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:52.495801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:52.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:25:52.507107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:02.541 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:02.579174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:02.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:02.608129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:12.710 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:12.747993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:12.722 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:12.760080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:22.782 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:22.819429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:22.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:22.889068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:32.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:32.999936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:32.976 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:33.012140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:43.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:43.209028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:43.184 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:43.219352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:53.251 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:53.285611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:53.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:26:53.297960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:03.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:03.416722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:03.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:03.429647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:13.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:13.523685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:13.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:13.535649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:23.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:23.599932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:23.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:23.611960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:33.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:33.706830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:33.686 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:33.718251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:43.758 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:43.789419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:43.769 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:43.800059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:53.869 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:53.900185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:53.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:27:53.913841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:03.961 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:03.990647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:03.974 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:04.004429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:14.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:14.108161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:14.091 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:14.121111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:24.157 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:24.185589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:24.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:24.197557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:34.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:34.296608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:34.281 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:34.309766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:44.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:44.402705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:44.386 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:44.414198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:54.461 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:54.487638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:54.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:28:54.501756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:04.565 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:04.590662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:04.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:04.601963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:14.683 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:14.709017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:14.696 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:14.721838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:24.791 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:24.816487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:24.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:24.830665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:34.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:34.935207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:34.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:34.948719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:44.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:45.015607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:45.006 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:45.030094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:55.105 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:55.128887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:55.126 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:29:55.145586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:05.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:05.210092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:05.199 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:05.222438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:15.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:15.313778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:15.305 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:15.326931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:25.368 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:25.390535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:25.381 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:25.403467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:35.530 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:35.550948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:35.545 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:35.566236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:45.638 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:45.659609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:45.652 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:45.673166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:55.767 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:55.786916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:55.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:30:55.800447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:05.892 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:05.912148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:05.904 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:05.924494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:16.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:16.076415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:16.069 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:16.088672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:26.149 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:26.167769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:26.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:26.236250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:36.325 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:36.343844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:36.337 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:36.355892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:46.412 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:46.429991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:46.424 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:46.442925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:56.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:56.587639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:56.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:31:56.602416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:06.705 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:06.722052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:06.716 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:06.733827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:16.821 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:16.837813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:16.834 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:16.851517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:26.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:26.924856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:26.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:26.938903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:37.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:37.099033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:37.094 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:37.110429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:47.210 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:47.225626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:47.222 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:47.238458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:57.287 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:57.302662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:57.300 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:32:57.315818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:07.417 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:07.432622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:07.429 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:07.444929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:17.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:17.594820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:17.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:17.605958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:27.690 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:27.704726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:27.702 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:27.716278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:37.802 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:37.815750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:37.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:37.826666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:47.896 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:47.910070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:47.909 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:47.922792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:58.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:58.028554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:58.030 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:33:58.043821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:08.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:08.147314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:08.152 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:08.163546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:18.290 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:18.302937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:18.303 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:18.315702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:28.413 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:28.425200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:28.424 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:28.436318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:38.506 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:38.517762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:38.521 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:38.533303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:48.604 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:48.616081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:48.616 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:48.627972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:58.689 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:58.700959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:58.698 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:34:58.710113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:08.837 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:08.848180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:08.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:08.863527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:18.917 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:18.928290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:18.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:18.940416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:29.047 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:29.057862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:29.061 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:29.072033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:39.128 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:39.137974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:39.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:39.193221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:49.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:49.301842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:49.306 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:49.316349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:59.380 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:59.389467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:59.437 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:35:59.446855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:09.515 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:09.524590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:09.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:09.538847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:19.652 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:19.661677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:19.665 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:19.674162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:29.812 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:29.820557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:29.823 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:29.832437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:39.971 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:39.979155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:39.984 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:39.992938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:50.048 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:50.056931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:50.124 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:36:50.133351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:00.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:00.221195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:00.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:00.234382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:10.288 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:10.295647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:10.299 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:10.307694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:20.422 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:20.430074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:20.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:20.442464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:24.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:24.794699 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:37:25.470 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:25.477305 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:37:25.467+0000 7f6fcbbda340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:37:25.471 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:37:25.467+0000 7f6fcbbda340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:37:25.471 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:37:25.467+0000 7f6fcbbda340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:37:25.471 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:37:30.534 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:30.541807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:30.548 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:30.555415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:40.610 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:40.617779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:40.624 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:40.631532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:50.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:50.686122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:50.743 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:37:50.750473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:00.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:00.842201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:00.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:00.852289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:10.963 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:10.969655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:10.977 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:10.984052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:21.043 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:21.049851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:21.056 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:21.063116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:31.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:31.126379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:31.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:31.140536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:41.195 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:41.201414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:41.209 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:41.215333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:51.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:51.321265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:51.326 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:38:51.332424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:01.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:01.403763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:01.410 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:01.415773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:11.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:11.483482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:11.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:11.497527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:21.563 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:21.567740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:21.576 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:21.581049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:31.707 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:31.711904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:31.720 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:31.725592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:41.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:41.797049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:41.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:41.810067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:51.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:51.917289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:51.924 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:39:51.929412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:02.043 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:02.047322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:02.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:02.060325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:12.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:12.160412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:12.169 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:12.173269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:22.235 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:22.239515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:22.248 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:22.252505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:32.358 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:32.361520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:32.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:32.393469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:42.465 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:42.469114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:42.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:42.483035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:52.544 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:52.547793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:52.557 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:40:52.560364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:02.617 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:02.620991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:02.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:02.678510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:12.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:12.747708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:12.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:12.808704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:22.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:22.875609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:22.884 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:22.887971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:32.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:32.983171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:32.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:32.992965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:43.092 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:43.094559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:43.106 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:43.109131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:53.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:53.188989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:53.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:41:53.263101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:03.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:03.351434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:03.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:03.365834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:13.477 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:13.479588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:13.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:13.494638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:23.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:23.594718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:23.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:23.608033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:33.726 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:33.728129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:33.737 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:33.739574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:43.811 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:43.812745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:43.823 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:43.825769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:53.899 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:53.900649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:53.955 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:42:53.956956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:04.027 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:04.028736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:04.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:04.040214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:14.162 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:14.163632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:14.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:14.175437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:24.316 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:24.317215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:24.327 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:24.328712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:34.435 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:34.436572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:34.445 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:34.446907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:44.738 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:44.739064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:44.749 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:44.750538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:54.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:54.849532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:54.861 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:43:54.862461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:05.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:05.134735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:05.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:05.147300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:15.266 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:15.266758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:15.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:15.276361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:25.364 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:25.364348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:25.438 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:25.438742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:35.550 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:35.550186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:35.562 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:35.562759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:45.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:45.677947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:45.705 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:45.705452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:55.788 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:55.788602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:55.801 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:44:55.801678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:05.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:05.910260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:05.925 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:05.925570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:15.984 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:15.984554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:15.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:15.995823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:26.074 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:26.073852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:26.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:26.183105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:36.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:36.338291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:36.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:36.349612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:46.452 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:46.451386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:46.464 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:46.463803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:56.550 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:56.550262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:56.562 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:45:56.562484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:06.694 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:06.693566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:06.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:06.718837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:16.820 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:16.819305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:16.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:16.851290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:26.967 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:26.966720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:26.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:26.979119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:37.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:37.119079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:37.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:37.133816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:47.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:47.245147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:47.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:47.257505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:57.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:57.377657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:57.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:46:57.389372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:07.456 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:07.454931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:07.469 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:07.468046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:17.557 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:17.555991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:17.569 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:17.568307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:27.645 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:27.643607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:27.659 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:27.657798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:37.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:37.771789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:37.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:37.785254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:47.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:47.901546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:47.913 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:47.912225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:58.063 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:58.061627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:58.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:47:58.073728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:08.246 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:08.244106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:08.257 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:08.255748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:18.323 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:18.321391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:18.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:18.336331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:28.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:28.469903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:28.484 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:28.481916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:38.554 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:38.552200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:38.565 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:38.563765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:48.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:48.651137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:48.665 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:48.663132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:58.755 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:58.752430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:58.769 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:48:58.767710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:08.838 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:08.835842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:08.854 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:08.852386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:18.965 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:18.962668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:18.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:18.977207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:29.049 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:29.046684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:29.061 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:29.059070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:39.177 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:39.174434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:39.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:39.185655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:49.284 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:49.281707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:49.295 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:49.293274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:59.402 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:59.400005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:59.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:49:59.414182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:09.485 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:09.482466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:09.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:09.537588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:19.660 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:19.658003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:19.674 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:19.672195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:29.782 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:29.779468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:29.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:29.794841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:39.914 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:39.911341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:39.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:39.925631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:50.058 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:50.055329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:50.069 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:50:50.066871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:00.139 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:00.135749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:00.150 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:00.147244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:10.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:10.255904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:10.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:10.267361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:20.332 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:20.329372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:20.344 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:20.341760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:30.471 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:30.468106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:30.482 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:30.479568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:40.548 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:40.545314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:40.603 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:40.600107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:50.671 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:50.667648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:50.682 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:51:50.679648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:00.756 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:00.752846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:00.769 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:00.766363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:10.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:10.852063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:10.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:10.869149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:20.956 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:20.953186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:20.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:20.967225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:31.052 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:31.048313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:31.064 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:31.061241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:41.207 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:41.203676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:41.220 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:41.216521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:51.329 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:51.324918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:51.342 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:52:51.338334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:01.460 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:01.456197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:01.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:01.469297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:11.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:11.538389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:11.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:11.552752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:21.649 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:21.644508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:21.660 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:21.656409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:31.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:31.768177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:31.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:31.783042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:41.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:41.847757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:41.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:41.918771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:51.996 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:51.991933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:52.009 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:53:52.005181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:02.121 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:02.115911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:02.134 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:02.128989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:05.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:05.485537 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T19:54:06.178 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:06.172409 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:54:06.161+0000 7fcc03c28340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:54:06.178 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:54:06.161+0000 7fcc03c28340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:54:06.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T19:54:06.161+0000 7fcc03c28340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:54:06.179 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:54:12.276 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:12.271814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:12.290 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:12.285964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:22.356 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:22.350930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:22.419 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:22.414762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:32.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:32.484813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:32.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:32.496203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:42.567 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:42.561824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:42.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:42.576070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:52.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:52.669676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:52.686 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:54:52.681302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:02.816 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:02.810360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:02.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:02.824138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:12.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:12.921779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:12.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:12.934578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:23.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:23.065427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:23.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:23.077904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:33.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:33.148948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:33.199 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:33.194005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:43.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:43.264243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:43.283 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:43.276399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:53.342 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:53.335911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:53.353 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:55:53.347987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:03.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:03.459666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:03.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:03.472254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:13.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:13.574916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:13.596 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:13.590077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:23.732 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:23.726293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:23.750 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:23.741264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:33.851 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:33.844530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:33.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:33.857593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:43.975 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:43.969033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:43.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:43.980873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:54.063 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:54.056948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:54.110 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:56:54.104252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:04.221 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:04.214971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:04.238 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:04.231530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:14.304 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:14.297514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:14.317 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:14.311199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:24.437 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:24.430504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:24.448 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:24.441580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:34.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:34.516160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:34.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:34.528910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:44.641 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:44.634117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:44.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:44.646539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:54.778 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:54.771341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:54.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:57:54.785852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:04.975 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:04.967858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:04.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:04.984767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:15.119 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:15.111882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:15.129 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:15.122458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:25.209 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:25.201803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:25.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:25.216063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:35.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:35.371677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:35.393 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:35.381949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:45.560 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:45.552913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:45.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:45.567509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:55.714 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:55.706343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:55.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:58:55.721447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:05.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:05.801720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:05.821 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:05.814001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:15.915 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:15.907011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:15.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:15.920044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:26.050 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:26.042498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:26.067 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:26.059833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:36.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:36.127294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:36.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:36.139779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:46.255 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:46.246601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:46.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:46.259828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:56.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:56.366955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:56.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 19:59:56.381013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:06.458 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:06.450146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:06.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:06.464171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:16.539 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:16.531257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:16.595 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:16.587638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:26.671 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:26.663790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:26.684 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:26.676288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:36.750 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:36.742606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:36.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:36.755141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:46.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:46.822800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:46.839 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:46.832558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:56.957 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:56.949790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:56.969 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:00:56.961901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:07.071 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:07.064466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:07.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:07.076555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:17.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:17.191255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:17.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:17.206503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:27.316 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:27.309402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:27.325 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:27.318226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:37.395 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:37.388601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:37.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:37.399647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:47.476 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:47.470116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:47.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:47.484685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:57.553 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:57.547450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:57.563 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:01:57.557810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:07.631 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:07.625731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:07.723 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:07.717415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:17.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:17.847023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:17.863 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:17.857555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:27.965 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:27.959191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:27.976 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:27.970674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:38.118 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:38.112754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:38.131 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:38.126236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:48.214 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:48.208961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:48.277 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:48.271774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:58.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:58.403600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:58.420 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:02:58.414897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:08.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:08.481195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:08.502 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:08.497474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:18.571 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:18.565888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:18.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:18.581550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:28.654 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:28.649928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:28.714 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:28.708943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:38.778 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:38.773978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:38.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:38.830114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:48.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:48.929782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:48.949 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:48.944566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:59.043 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:59.039100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:59.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:03:59.052827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:09.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:09.151010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:09.171 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:09.167514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:19.242 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:19.237771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:19.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:19.248655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:29.333 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:29.329684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:29.436 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:29.432163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:39.553 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:39.549338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:39.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:39.564702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:49.636 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:49.632743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:49.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:49.646696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:59.755 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:59.751654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:59.766 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:04:59.762978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:09.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:09.831138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:09.882 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:09.878552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:19.997 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:19.993805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:20.009 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:20.006461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:30.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:30.073819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:30.087 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:30.084285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:40.195 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:40.191904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:40.210 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:40.207137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:50.280 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:50.276815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:50.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:05:50.290830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:00.360 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:00.356786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:00.424 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:00.421944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:10.536 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:10.532808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:10.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:10.544963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:20.662 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:20.659242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:20.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:20.672647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:30.819 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:30.816352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:30.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:30.827843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:40.946 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:40.943492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:40.959 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:40.956586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:51.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:51.052718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:51.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:06:51.068067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:01.200 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:01.196994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:01.211 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:01.209307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:11.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:11.260216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:11.276 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:11.274006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:21.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:21.397159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:21.410 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:21.408430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:31.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:31.519815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:31.533 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:31.531272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:41.610 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:41.607802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:41.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:41.666088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:51.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:51.789388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:51.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:07:51.802294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:01.891 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:01.889398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:01.905 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:01.903064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:12.044 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:12.042275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:12.056 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:12.054274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:22.149 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:22.147770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:22.211 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:22.209730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:32.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:32.290144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:32.305 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:32.303431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:42.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:42.377502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:42.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:42.388437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:52.462 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:52.460587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:52.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:08:52.473174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:02.567 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:02.565490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:02.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:02.579455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:12.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:12.645994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:12.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:12.674504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:22.783 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:22.781723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:22.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:22.795374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:32.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:32.903724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:32.919 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:32.918102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:42.994 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:42.992118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:43.006 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:43.004822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:53.087 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:53.086002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:53.101 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:09:53.099531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:03.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:03.166845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:03.180 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:03.179121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:13.280 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:13.278887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:13.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:13.292203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:23.377 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:23.375493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:23.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:23.387503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:33.497 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:33.495797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:33.509 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:33.508214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:43.651 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:43.649404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:43.662 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:43.660903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:46.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:46.182265 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T20:10:46.856 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:46.854194 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:10:46.845+0000 7f476e8b1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:10:46.856 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:10:46.845+0000 7f476e8b1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:10:46.857 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:10:46.845+0000 7f476e8b1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:10:46.857 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:10:53.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:53.909519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:53.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:10:53.921943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:03.986 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:03.985191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:04.000 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:04.000104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:14.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:14.069504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:14.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:14.144739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:24.243 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:24.242056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:24.254 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:24.253575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:34.357 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:34.356520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:34.370 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:34.369065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:44.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:44.452429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:44.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:44.466279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:54.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:54.535994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:54.549 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:11:54.548556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:04.606 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:04.605152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:04.618 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:04.617383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:14.677 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:14.676865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:14.721 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:14.720907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:24.869 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:24.868707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:24.878 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:24.878059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:35.032 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:35.031089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:35.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:35.044616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:45.104 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:45.103816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:45.117 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:45.117233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:55.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:55.185073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:55.204 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:12:55.199419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:05.352 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:05.351741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:05.370 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:05.369911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:15.438 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:15.438129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:15.488 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:15.488172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:25.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:25.591206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:25.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:25.604500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:35.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:35.673128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:35.685 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:35.684862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:45.767 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:45.766230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:45.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:45.779854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:55.898 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:55.897967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:55.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:13:55.910502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:05.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:05.987026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:05.999 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:05.999149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:16.069 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:16.068687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:16.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:16.080575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:26.144 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:26.143342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:26.156 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:26.156042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:36.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:36.229390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:36.241 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:36.241539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:46.330 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:46.329185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:46.344 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:46.344087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:56.442 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:56.441712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:56.454 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:14:56.454072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:06.573 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:06.573647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:06.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:06.586607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:16.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:16.654867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:16.749 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:16.749635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:26.826 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:26.825648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:26.838 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:26.838146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:36.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:36.907784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:36.922 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:36.921783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:47.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:47.014848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:47.027 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:47.026879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:57.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:57.136333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:57.152 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:15:57.151734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:07.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:07.226962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:07.241 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:07.241122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:17.366 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:17.366097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:17.379 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:17.378819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:27.436 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:27.436576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:27.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:27.449403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:37.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:37.564599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:37.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:37.578778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:47.643 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:47.642639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:47.656 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:47.656497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:57.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:57.762794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:57.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:16:57.774134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:07.877 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:07.877033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:07.888 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:07.889040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:17.981 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:17.981015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:17.996 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:17.996153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:28.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:28.078473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:28.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:28.135568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:38.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:38.205716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:38.227 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:38.227178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:48.331 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:48.331768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:48.343 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:48.343810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:58.447 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:58.447647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:58.461 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:17:58.461991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:08.561 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:08.561036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:08.572 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:08.572967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:18.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:18.646055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:18.700 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:18.700238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:28.806 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:28.806188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:28.816 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:28.816719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:38.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:38.934305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:38.948 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:38.948548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:49.101 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:49.100768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:49.112 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:49.112677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:59.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:59.186502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:59.201 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:18:59.202117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:09.305 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:09.304816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:09.318 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:09.318302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:19.419 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:19.419294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:19.432 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:19.433033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:29.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:29.500866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:29.559 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:29.559702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:39.686 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:39.685613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:39.695 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:39.695517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:49.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:49.810183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:49.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:49.825339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:59.904 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:59.904112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:59.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:19:59.916510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:09.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:09.988521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:10.035 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:10.035874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:20.142 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:20.142710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:20.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:20.154886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:30.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:30.275562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:30.286 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:30.286806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:40.399 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:40.399672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:40.413 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:40.413823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:50.483 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:50.483533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:50.497 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:20:50.497255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:00.600 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:00.601021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:00.608 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:00.609337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:10.676 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:10.676434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:10.690 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:10.690802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:20.801 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:20.801830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:20.811 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:20.811432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:30.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:30.929625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:30.963 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:30.963939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:41.032 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:41.032725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:41.046 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:41.046239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:51.118 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:51.118672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:51.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:21:51.189592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:01.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:01.253340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:01.265 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:01.265867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:11.385 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:11.385434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:11.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:11.396828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:21.470 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:21.470274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:21.483 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:21.483564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:31.553 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:31.553737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:31.614 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:31.613535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:41.737 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:41.737251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:41.751 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:41.751929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:51.817 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:51.817383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:51.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:22:51.829341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:01.928 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:01.928473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:01.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:01.941216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:12.040 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:12.040876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:12.050 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:12.050047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:22.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:22.173613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:22.184 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:22.184275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:32.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:32.282262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:32.295 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:32.295478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:42.369 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:42.369487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:42.423 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:42.423346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:52.505 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:52.505646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:52.516 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:23:52.516698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:02.616 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:02.616412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:02.627 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:02.627528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:12.771 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:12.771299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:12.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:12.784144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:22.852 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:22.852213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:22.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:22.902606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:33.031 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:33.031168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:33.046 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:33.046130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:43.137 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:43.137416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:43.150 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:43.150987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:53.245 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:53.245073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:53.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:24:53.260434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:03.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:03.375254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:03.388 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:03.389075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:13.488 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:13.488746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:13.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:13.504070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:23.639 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:23.638896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:23.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:23.650255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:33.798 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:33.798777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:33.811 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:33.812025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:43.935 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:43.935588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:43.948 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:43.948555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:54.011 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:54.011636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:54.024 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:25:54.024968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:04.096 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:04.096933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:04.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:04.109156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:14.180 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:14.180435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:14.193 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:14.193845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:24.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:24.315100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:24.327 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:24.327670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:34.433 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:34.433488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:34.446 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:34.446459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:44.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:44.556108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:44.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:44.569009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:54.631 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:54.631546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:54.641 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:26:54.641908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:04.788 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:04.788412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:04.806 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:04.801930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:14.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:14.915987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:14.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:14.929538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:25.024 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:25.024654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:25.034 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:25.035135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:26.860 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:26.860402 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T20:27:27.546 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:27.546156 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:27:27.535+0000 7fce6e2dd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:27:27.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:27:27.535+0000 7fce6e2dd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:27:27.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:27:27.535+0000 7fce6e2dd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:27:27.548 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:27:35.113 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:35.113119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:35.125 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:35.125595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:45.187 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:45.187819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:45.201 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:45.201669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:55.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:55.261494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:55.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:27:55.275834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:05.404 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:05.403640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:05.417 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:05.416819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:15.477 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:15.477437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:15.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:15.489235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:25.604 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:25.603843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:25.616 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:25.616582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:35.679 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:35.678603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:35.690 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:35.689837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:45.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:45.763875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:45.776 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:45.776088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:55.846 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:55.845844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:55.861 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:28:55.861440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:05.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:05.964943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:05.983 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:05.983306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:16.093 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:16.093063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:16.105 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:16.105539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:26.174 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:26.173805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:26.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:26.187756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:36.328 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:36.327807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:36.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:36.337813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:46.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:46.472415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:46.484 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:46.483630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:56.595 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:56.594247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:56.607 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:29:56.606524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:06.667 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:06.665871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:06.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:06.679128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:16.806 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:16.804160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:16.822 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:16.821315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:26.922 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:26.920982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:26.940 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:26.936071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:36.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:36.993856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:37.008 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:37.007083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:47.072 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:47.070842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:47.084 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:47.083201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:57.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:57.153408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:57.244 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:30:57.241936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:07.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:07.358532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:07.376 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:07.374102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:17.450 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:17.448283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:17.465 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:17.463107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:27.572 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:27.569757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:27.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:27.584170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:37.696 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:37.694072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:37.716 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:37.710228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:47.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:47.771279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:47.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:47.785614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:57.856 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:57.853782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:57.866 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:31:57.863859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:07.971 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:07.968238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:07.990 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:07.983007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:18.049 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:18.042459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:18.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:18.054877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:28.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:28.117567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:28.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:28.176492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:38.274 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:38.271351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:38.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:38.289885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:48.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:48.393929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:48.408 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:48.405883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:58.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:58.533484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:58.551 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:32:58.547976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:08.654 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:08.650842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:08.666 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:08.663583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:18.735 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:18.731738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:18.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:18.805576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:28.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:28.899561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:28.915 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:28.912281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:38.968 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:38.964928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:38.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:38.974885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:49.076 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:49.071902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:49.089 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:49.085697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:59.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:59.168767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:59.186 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:33:59.182456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:09.299 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:09.294982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:09.311 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:09.307997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:19.386 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:19.382555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:19.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:19.395283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:29.483 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:29.473743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:29.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:29.486133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:39.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:39.622786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:39.639 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:39.636345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:49.718 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:49.715693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:49.772 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:49.770463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:59.892 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:59.889703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:59.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:34:59.904622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:10.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:10.204464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:10.220 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:10.218414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:20.358 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:20.356721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:20.370 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:20.369053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:30.552 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:30.550704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:30.566 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:30.564753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:40.634 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:40.633345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:40.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:40.645943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:50.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:50.761225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:50.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:35:50.773771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:00.892 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:00.891641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:00.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:00.906080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:10.974 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:10.974198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:10.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:10.987290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:21.058 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:21.058027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:21.072 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:21.072248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:31.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:31.197080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:31.215 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:31.212231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:41.332 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:41.333282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:41.346 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:41.346847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:51.417 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:51.418165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:51.427 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:36:51.428202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:01.481 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:01.482011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:01.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:01.493708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:11.585 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:11.586450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:11.598 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:11.598688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:21.740 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:21.742265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:21.754 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:21.756256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:31.867 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:31.869284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:31.882 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:31.884937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:42.011 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:42.013267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:42.022 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:42.024881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:52.086 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:52.088550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:52.103 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:37:52.106140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:02.193 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:02.195593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:02.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:02.208881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:12.326 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:12.328791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:12.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:12.341839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:22.421 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:22.424598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:22.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:22.437464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:32.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:32.526128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:32.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:32.538959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:42.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:42.679033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:42.688 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:42.692363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:52.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:52.789443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:52.851 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:38:52.854917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:02.951 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:02.954643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:02.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:02.968298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:13.054 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:13.058347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:13.066 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:13.070872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:23.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:23.152314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:23.160 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:23.164071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:33.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:33.262646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:33.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:33.275203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:43.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:43.388386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:43.393 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:43.398108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:53.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:53.475793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:53.498 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:39:53.503284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:03.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:03.572673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:03.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:03.586971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:13.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:13.701470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:13.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:13.724969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:23.845 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:23.850185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:23.856 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:23.862039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:33.935 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:33.940437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:33.950 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:33.955861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:44.024 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:44.029900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:44.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:44.042951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:54.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:54.142041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:54.151 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:40:54.157252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:04.242 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:04.247770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:04.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:04.258822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:14.380 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:14.386203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:14.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:14.395066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:24.480 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:24.486173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:24.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:24.498433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:34.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:34.576768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:34.582 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:34.588805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:44.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:44.653000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:44.660 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:44.666942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:54.731 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:54.738341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:54.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:41:54.751121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:04.815 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:04.821687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:04.827 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:04.834461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:14.907 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:14.914027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:14.973 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:14.980709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:25.095 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:25.101800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:25.109 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:25.116439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:35.178 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:35.185856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:35.192 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:35.200323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:45.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:45.265687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:45.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:45.278453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:55.376 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:55.381218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:55.391 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:42:55.398712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:05.462 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:05.470182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:05.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:05.481078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:15.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:15.598695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:15.604 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:15.612498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:25.671 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:25.679024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:25.731 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:25.739826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:35.860 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:35.868175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:35.871 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:35.879212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:45.950 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:45.958391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:45.960 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:45.969164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:56.099 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:56.107737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:56.111 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:43:56.120107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:06.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:06.238255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:06.242 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:06.251281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:07.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:07.555646 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T20:44:08.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:08.214610 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:44:08.206+0000 7fd0badd6340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:44:08.207 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:44:08.206+0000 7fd0badd6340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:44:08.207 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T20:44:08.206+0000 7fd0badd6340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:44:08.207 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:44:16.302 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:16.310999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:16.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:16.324244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:26.388 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:26.397042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:26.400 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:26.409307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:36.470 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:36.479338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:36.530 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:36.539105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:46.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:46.599922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:46.601 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:46.610362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:56.700 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:56.709212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:56.713 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:44:56.722533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:06.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:06.795036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:06.798 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:06.807648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:16.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:16.912347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:16.914 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:16.923573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:26.978 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:26.987787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:26.989 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:26.998525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:37.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:37.117530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:37.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:37.130152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:47.239 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:47.248472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:47.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:47.262466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:57.381 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:57.390621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:57.392 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:45:57.402371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:07.543 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:07.553307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:07.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:07.566339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:17.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:17.636083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:17.637 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:17.647385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:27.755 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:27.765145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:27.765 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:27.775316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:37.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:37.911674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:37.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:37.926393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:48.035 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:48.045336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:48.047 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:48.057390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:58.161 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:58.171690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:58.175 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:46:58.185698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:08.266 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:08.276581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:08.279 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:08.289686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:18.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:18.407328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:18.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:18.420412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:28.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:28.645334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:28.688 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:28.698792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:38.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:38.771625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:38.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:38.783548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:48.901 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:48.911873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:48.914 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:48.925594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:59.008 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:59.018868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:59.020 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:47:59.031132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:09.151 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:09.162132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:09.176 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:09.185776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:19.288 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:19.298653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:19.299 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:19.310623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:29.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:29.406793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:29.410 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:29.420995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:39.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:39.534028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:39.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:39.552811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:49.620 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:49.631502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:49.633 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:49.644597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:59.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:59.797966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:59.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:48:59.810200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:09.888 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:09.899178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:09.899 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:09.910348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:20.024 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:20.033145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:20.036 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:20.046984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:30.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:30.147347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:30.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:30.158558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:40.281 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:40.292411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:40.294 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:40.305390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:50.413 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:50.424767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:50.430 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:49:50.441531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:00.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:00.551105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:00.551 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:00.562305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:10.618 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:10.629668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:10.627 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:10.639453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:20.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:20.773424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:20.775 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:20.786641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:30.871 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:30.882967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:30.886 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:30.898029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:40.966 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:40.977938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:40.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:40.992302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:51.060 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:51.071603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:51.142 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:50:51.154060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:01.214 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:01.225419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:01.278 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:01.289860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:11.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:11.395243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:11.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:11.410023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:21.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:21.489676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:21.496 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:21.503424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:31.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:31.616457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:31.641 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:31.652445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:41.743 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:41.755099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:41.756 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:41.768545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:51.849 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:51.860881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:51.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:51:51.874025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:01.934 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:01.946366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:01.947 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:01.958739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:12.023 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:12.035257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:12.035 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:12.047298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:22.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:22.119914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:22.121 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:22.133076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:32.192 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:32.203868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:32.232 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:32.244085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:42.301 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:42.313276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:42.314 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:42.326186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:52.444 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:52.456171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:52.456 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:52:52.468834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:02.572 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:02.583801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:02.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:02.592927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:12.696 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:12.708412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:12.710 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:12.722269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:22.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:22.798112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:22.798 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:22.810659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:32.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:32.909182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:32.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:32.925013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:42.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:42.992951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:42.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:43.007569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:53.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:53.132364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:53.130 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:53:53.142442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:03.209 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:03.221749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:03.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:03.235692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:13.334 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:13.346054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:13.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:13.361838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:23.462 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:23.475063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:23.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:23.487951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:33.573 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:33.585649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:33.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:33.598530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:43.661 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:43.674062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:43.673 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:43.685802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:53.777 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:53.789880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:53.791 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:54:53.803643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:03.865 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:03.877662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:03.874 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:03.887015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:13.985 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:13.997359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:14.001 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:14.012306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:24.103 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:24.116311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:24.116 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:24.129014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:34.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:34.194652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:34.193 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:34.206338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:44.319 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:44.331323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:44.331 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:44.343952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:54.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:54.591354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:54.594 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:55:54.606914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:04.747 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:04.759272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:04.759 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:04.771796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:14.849 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:14.861635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:14.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:14.874853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:24.981 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:24.993705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:24.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:25.004449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:35.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:35.083314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:35.130 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:35.143212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:45.198 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:45.211159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:45.269 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:45.282259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:55.381 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:55.394284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:55.392 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:56:55.405704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:05.569 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:05.582332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:05.578 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:05.591237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:15.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:15.724385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:15.725 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:15.737828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:25.888 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:25.901544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:25.902 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:25.915492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:36.005 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:36.017682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:36.019 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:36.032607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:46.098 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:46.111110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:46.110 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:46.123252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:56.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:56.203142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:56.203 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:57:56.216501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:06.302 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:06.314789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:06.318 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:06.331511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:16.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:16.428894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:16.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:16.447246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:26.556 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:26.568846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:26.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:26.584080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:36.688 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:36.701326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:36.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:36.712190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:46.781 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:46.793364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:46.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:46.805564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:56.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:56.896147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:56.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:58:56.909007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:06.961 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:06.974433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:06.974 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:06.987487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:17.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:17.050684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:17.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:17.069733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:27.167 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:27.180268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:27.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:27.192653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:37.266 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:37.279661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:37.278 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:37.291738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:47.371 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:47.384847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:47.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:47.396693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:57.451 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:57.464559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:57.508 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 20:59:57.521822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:07.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:07.592590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:07.593 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:07.606972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:17.732 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:17.745553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:17.745 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:17.759111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:27.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:27.910292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:27.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:27.921360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:37.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:37.980916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:38.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:38.088327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:48.207 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:48.220056 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T21:00:48.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:48.220457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:48.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:48.232036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:48.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:48.842955 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T21:00:48.832+0000 7fe23e217340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T21:00:48.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:00:48.832+0000 7fe23e217340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T21:00:48.832 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:00:48.832+0000 7fe23e217340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T21:00:48.832 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T21:00:58.308 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:58.318143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:58.322 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:00:58.330941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:08.412 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:08.425947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:08.428 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:08.436894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:18.521 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:18.533796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:18.532 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:18.545031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:28.615 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:28.623578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:28.624 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:28.635805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:38.731 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:38.740620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:38.757 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:38.754533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:48.851 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:48.864269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:48.862 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:48.875370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:58.959 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:58.969686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:58.975 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:01:58.980154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:09.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:09.055606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:09.057 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:09.068409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:19.117 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:19.129504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:19.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:19.195089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:29.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:29.272056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:29.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:29.284169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:39.344 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:39.356920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:39.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:39.367231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:49.463 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:49.475229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:49.504 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:49.487317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:59.551 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:59.559915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:59.591 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:02:59.574163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:09.782 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:09.789283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:09.805 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:09.803517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:19.885 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:19.898469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:19.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:19.910985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:29.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:30.005616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:30.005 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:30.017445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:40.201 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:40.132840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:40.202 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:40.146470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:50.251 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:50.249973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:50.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:03:50.265382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:00.348 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:00.334213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:00.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:00.349107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:10.396 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:10.406181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:10.405 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:10.417188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:20.519 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:20.530874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:20.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:20.544584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:30.658 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:30.671583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:30.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:30.684364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:40.790 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:40.803341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:40.807 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:40.816121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:50.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:50.877142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:50.884 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:04:50.892460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:00.975 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:00.985988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:00.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:01.000280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:11.053 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:11.063988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:11.068 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:11.077151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:21.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:21.175802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:21.174 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:21.185162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:31.367 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:31.380648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:31.432 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:31.446401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:41.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:41.548863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:41.550 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:41.560437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:51.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:51.725466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:51.722 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:05:51.736321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:01.830 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:01.844322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:01.871 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:01.885084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:11.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:11.955331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:11.958 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:11.968089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:22.026 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:22.039842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:22.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:22.055037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:32.115 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:32.128857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:32.129 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:32.141932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:42.199 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:42.212402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:42.211 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:42.225039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:52.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:52.306262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:52.304 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:06:52.317671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:02.407 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:02.420905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:02.422 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:02.434884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:12.527 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:12.540522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:12.538 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:12.551807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:22.629 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:22.643529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:22.642 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:22.656168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:32.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:32.775158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:32.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:32.787217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:42.865 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:42.879454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:42.880 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:42.893957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:53.011 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:53.025141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:53.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:07:53.056116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:03.214 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:03.227853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:03.233 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:03.247291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:13.342 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:13.355796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:13.354 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:13.368173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:23.445 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:23.458527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:23.459 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:23.472789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:33.548 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:33.562170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:33.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:33.577955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:43.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:43.688289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:43.689 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:43.702995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:53.753 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:53.766579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:08:53.764 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:08:53.777278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:03.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:03.877661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:03.877 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:03.890711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:13.937 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:13.949798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:13.950 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:13.963541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:24.031 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:24.043418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:24.047 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:24.055705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:34.130 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:34.142156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:34.183 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:34.195237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:44.291 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:44.303508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:44.381 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:44.316306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:54.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:54.418280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:09:54.420 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:09:54.432152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:04.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:04.498116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:04.496 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:04.508336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:14.561 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:14.573010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:14.576 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:14.588042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:24.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:24.710328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:24.728 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:24.739845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:34.900 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:34.911093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:34.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:34.923196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:45.053 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:45.063890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:45.070 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:45.081022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:55.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:55.198845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:10:55.200 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:10:55.211768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:05.335 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:05.346473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:05.354 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:05.365596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:15.456 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:15.466580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:15.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:15.477253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:25.573 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:25.583632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:25.585 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:25.596126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:35.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:35.758912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:35.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:35.771818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:45.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:45.839381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:45.842 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:45.852380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:55.945 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:55.955241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:11:55.956 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:11:55.966523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:06.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:06.087797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:06.091 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:06.101237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:16.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:16.172429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:16.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:16.183632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:26.240 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:26.250211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:26.307 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:26.316643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:36.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:36.418394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:36.419 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:36.429430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:46.529 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:46.538089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:46.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:46.550211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:56.658 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:56.667357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:12:56.672 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:12:56.681930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:06.784 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:06.793083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:06.795 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:06.804639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:16.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:16.906679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:16.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:16.918952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:27.014 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:27.022689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:27.023 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:27.032703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:37.128 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:37.136891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:37.141 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:37.149952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:47.216 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:47.224523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:47.283 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:47.291756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:57.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:57.417529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:13:57.421 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:13:57.429814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:07.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:07.526955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:07.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:07.540123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:17.605 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:17.613542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:17.618 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:17.626736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:27.683 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:27.690956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:27.695 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:27.703333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:37.762 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:37.770334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:37.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:37.782561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:47.896 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:47.903828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:47.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:47.916118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:58.025 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:58.033148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:14:58.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:14:58.046537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:08.164 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:08.171737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:08.177 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:08.185175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:18.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:18.277902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:18.288 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:18.296074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:28.368 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:28.375379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:28.382 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:28.389853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:38.456 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:38.463975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:38.468 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:38.475743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:48.576 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:48.583002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:48.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:48.596518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:58.685 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:58.692261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:15:58.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:15:58.705109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:08.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:08.803331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:08.808 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:08.815841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:18.913 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:18.919783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:18.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:18.934158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:29.061 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:29.068631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:29.074 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:29.080980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:39.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:39.266462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:39.281 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:39.282845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:49.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:49.369825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:49.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:49.422582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:59.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:59.496153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:16:59.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:16:59.509938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:09.613 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:09.617166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:09.624 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:09.630749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:19.699 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:19.705801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:19.712 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:19.718555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:28.847 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:28.854018 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T21:17:29.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:29.537281 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T21:17:29.523+0000 7fe2afe45340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T21:17:29.531 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:17:29.523+0000 7fe2afe45340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T21:17:29.532 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:17:29.523+0000 7fe2afe45340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T21:17:29.532 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T21:17:29.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:29.782039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:29.787 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:29.793578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:39.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:39.908967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:39.915 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:39.921567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:50.020 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:50.026673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:17:50.042 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:17:50.042655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:00.089 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:00.095204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:00.139 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:00.145515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:10.260 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:10.266900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:10.273 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:10.279924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:20.398 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:20.403846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:20.408 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:20.413513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:30.473 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:30.478998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:30.481 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:30.487582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:40.623 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:40.628782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:40.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:40.641549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:50.745 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:50.751240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:18:50.757 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:18:50.763074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:00.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:00.830856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:00.839 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:00.844878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:10.899 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:10.905324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:10.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:10.918609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:21.052 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:21.058170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:21.065 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:21.071080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:31.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:31.202040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:31.210 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:31.216060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:41.275 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:41.280540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:41.287 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:41.292583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:51.358 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:51.364187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:19:51.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:19:51.377859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:01.491 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:01.496580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:01.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:01.508924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:11.572 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:11.578108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:11.584 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:11.590146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:21.653 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:21.658702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:21.664 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:21.670132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:31.734 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:31.739283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:31.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:31.754299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:41.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:41.860607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:41.867 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:41.872629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:51.979 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:51.984339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:20:51.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:20:51.997308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:02.102 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:02.107126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:02.115 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:02.121084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:12.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:12.233454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:12.238 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:12.244282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:22.348 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:22.353536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:22.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:22.369237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:32.505 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:32.510606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:32.518 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:32.523051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:42.664 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:42.669343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:42.675 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:42.680386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:52.738 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:52.743323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:21:52.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:21:52.797443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:02.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:02.888191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:02.951 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:02.956247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:13.016 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:13.021383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:13.030 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:13.035255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:23.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:23.138073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:23.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:23.151778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:33.261 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:33.265605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:33.273 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:33.278780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:43.369 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:43.374521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:43.386 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:43.390859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:53.529 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:53.534459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:22:53.539 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:22:53.544971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:03.627 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:03.631816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:03.642 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:03.647279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:13.740 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:13.745599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:13.757 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:13.762870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:23.917 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:23.922532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:23.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:23.934632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:33.985 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:33.990237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:33.998 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:34.003078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:44.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:44.083258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:44.090 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:44.094866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:54.201 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:54.206590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:23:54.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:23:54.218770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:04.324 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:04.328844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:04.335 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:04.340362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:14.425 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:14.429963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:14.440 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:14.444663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:24.509 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:24.514198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:24.521 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:24.526374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:34.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:34.659789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:34.667 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:34.672170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:44.766 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:44.770360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:44.778 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:44.782924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:54.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:54.875351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:24:54.883 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:24:54.888731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:04.956 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:04.961127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:04.972 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:04.976995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:15.069 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:15.073093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:15.082 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:15.086711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:25.149 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:25.153516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:25.162 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:25.167115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:35.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:35.262229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:35.271 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:35.276170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:45.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:45.354179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:45.359 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:45.364341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:55.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:55.483299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:25:55.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:25:55.493814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:05.588 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:05.592206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:05.601 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:05.605401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:15.715 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:15.719961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:15.727 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:15.732066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:25.822 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:25.826151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:25.833 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:25.837677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:35.895 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:35.899366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:35.908 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:35.913250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:46.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:46.011771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:46.022 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:46.027097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:56.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:56.137601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:26:56.152 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:26:56.155563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:06.248 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:06.251973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:06.262 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:06.266634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:16.330 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:16.334269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:16.341 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:16.345824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:26.441 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:26.445218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:26.453 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:26.457330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:36.523 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:36.526843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:36.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:36.539588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:46.650 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:46.653898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:46.662 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:46.666961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:56.794 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:56.798700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:27:56.811 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:27:56.815667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:06.891 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:06.895666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:06.911 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:06.915168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:17.039 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:17.043329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:17.051 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:17.055110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:27.162 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:27.166306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:27.175 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:27.179415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:37.245 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:37.248905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:37.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:37.262309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:47.365 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:47.366205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:47.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:47.382552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:57.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:57.476632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:28:57.495 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:28:57.490513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:07.568 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:07.571604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:07.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:07.583373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:17.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:17.813085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:17.822 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:17.825967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:27.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:27.914485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:27.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:27.973798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:38.235 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:38.238708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:38.247 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:38.251302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:48.342 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:48.345688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:48.354 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:48.357982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:58.429 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:58.433474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:29:58.443 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:29:58.447057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:08.549 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:08.553425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:08.563 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:08.567213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:18.630 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:18.633843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:18.643 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:18.647143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:28.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:28.751744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:28.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:28.764864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:38.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:38.834961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:38.890 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:38.894282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:48.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:48.991926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:49.003 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:49.006774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:59.122 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:59.125881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:30:59.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:30:59.139735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:09.241 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:09.245389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:09.255 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:09.258444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:19.349 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:19.351442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:19.359 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:19.363374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:29.506 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:29.510119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:29.517 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:29.521212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:39.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:39.590075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:39.597 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:39.601400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:49.711 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:49.714602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:49.724 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:49.727886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:59.804 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:59.808285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:31:59.817 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:31:59.821507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:09.931 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:09.934401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:09.944 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:09.948181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:20.017 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:20.020892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:20.032 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:20.035827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:30.115 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:30.119085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:30.125 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:30.129331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:40.225 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:40.228849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:40.233 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:40.237260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:50.330 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:50.333389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:32:50.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:32:50.342365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:00.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:00.412812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:00.421 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:00.425242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:10.549 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:10.553118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:10.561 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:10.565446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:20.672 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:20.675896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:20.684 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:20.687764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:30.774 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:30.777817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:30.853 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:30.856447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:40.950 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:40.953991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:40.962 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:40.966440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:51.039 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:51.042247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:33:51.053 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:33:51.057162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:01.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:01.166619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:01.182 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:01.185591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:09.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:09.550321 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T21:34:10.208 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:10.209966 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T21:34:10.200+0000 7fd6c6088340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T21:34:10.208 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:34:10.200+0000 7fd6c6088340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T21:34:10.209 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:34:10.200+0000 7fd6c6088340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T21:34:10.209 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T21:34:11.250 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:11.254222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:11.263 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:11.267590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:21.363 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:21.366187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:21.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:21.376505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:31.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:31.478742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:31.489 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:31.492977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:41.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:41.567359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:41.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:41.583451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:51.651 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:51.654204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:34:51.664 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:34:51.667849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:01.740 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:01.743307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:01.753 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:01.756701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:11.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:11.867887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:11.881 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:11.885091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:21.989 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:21.991623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:22.138 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:22.004063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:32.108 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:32.111794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:32.120 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:32.124333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:42.219 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:42.222104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:42.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:42.233778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:52.341 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:52.344821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:35:52.352 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:35:52.356178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:02.654 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:02.657297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:02.720 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:02.723375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:12.791 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:12.794751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:12.801 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:12.804723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:23.008 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:23.011510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:23.019 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:23.023019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:33.313 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:33.317125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:33.323 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:33.327367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:43.517 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:43.520927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:43.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:43.593176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:53.674 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:53.677962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:36:53.690 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:36:53.693742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:03.778 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:03.781223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:03.790 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:03.794105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:13.899 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:13.902728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:13.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:13.914230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:24.026 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:24.028978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:24.038 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:24.041879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:34.139 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:34.142514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:34.150 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:34.153983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:44.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:44.259440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:44.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:44.271809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:54.331 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:54.334711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:37:54.348 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:37:54.350871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:04.467 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:04.469819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:04.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:04.482960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:14.600 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:14.603068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:14.611 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:14.614664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:24.756 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:24.759741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:24.767 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:24.770460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:34.882 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:34.885332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:34.895 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:34.898484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:44.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:44.983792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:44.994 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:44.997417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:55.094 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:55.097954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:38:55.106 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:38:55.109945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:05.173 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:05.176192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:05.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:05.188896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:15.259 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:15.262100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:15.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:15.273739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:25.340 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:25.343069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:25.365 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:25.368383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:35.434 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:35.437216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:35.446 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:35.450065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:45.509 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:45.512389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:45.521 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:45.524383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:55.588 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:55.591158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:39:55.652 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:39:55.655442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:05.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:05.721750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:05.733 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:05.736050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:15.855 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:15.857779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:15.866 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:15.869768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:25.939 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:25.942297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:25.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:25.993876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:36.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:36.139715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:36.147 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:36.150591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:46.256 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:46.256747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:46.267 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:46.270211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:56.403 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:56.406701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:40:56.415 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:40:56.418649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:06.547 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:06.550293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:06.560 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:06.563338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:16.668 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:16.671640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:16.682 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:16.685025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:26.795 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:26.797929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:26.810 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:26.812563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:36.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:36.872919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:36.880 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:36.883604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:46.954 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:46.957542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:46.963 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:46.966925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:57.026 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:57.029252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:41:57.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:41:57.043817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:07.123 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:07.124927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:07.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:07.136867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:17.296 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:17.299739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:17.311 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:17.314105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:27.414 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:27.416712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:27.426 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:27.429538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:37.555 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:37.557733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:37.567 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:37.570216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:47.693 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:47.695607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:47.717 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:47.715229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:57.773 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:57.776114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:42:57.785 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:42:57.788273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:07.849 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:07.851745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:07.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:07.900274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:17.994 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:17.996673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:18.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:18.010357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:28.179 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:28.181355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:28.191 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:28.193482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:38.329 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:38.331723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:38.342 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:38.344465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:48.468 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:48.470444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:48.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:48.481706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:58.563 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:58.565969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:43:58.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:43:58.580045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:08.651 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:08.653230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:08.664 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:08.666728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:18.780 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:18.781513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:18.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:18.794387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:28.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:28.908551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:28.918 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:28.920602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:38.982 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:38.984470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:38.996 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:38.998609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:49.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:49.148210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:49.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:49.161972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:59.235 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:59.236929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:44:59.248 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:44:59.249697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:09.326 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:09.328087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:09.372 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:09.373926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:19.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:19.474139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:19.485 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:19.486964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:29.559 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:29.560764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:29.572 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:29.571873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:39.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:39.749624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:39.759 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:39.761608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:49.889 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:49.890443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:45:49.900 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:45:49.902272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:00.055 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:00.055844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:00.068 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:00.070098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:10.133 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:10.134918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:10.146 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:10.147383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:20.211 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:20.213021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:20.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:20.227677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:30.331 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:30.332153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:30.345 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:30.345970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:40.462 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:40.463718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:40.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:40.476436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:50.537 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:50.538791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:46:50.553 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:46:50.551198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:00.621 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:00.622028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:00.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:00.655832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:10.784 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:10.785566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:10.799 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:10.800598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:20.901 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:20.902899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:20.916 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:20.917347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:31.018 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:31.018911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:31.030 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:31.031986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:41.135 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:41.136360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:41.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:41.149198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:51.223 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:51.223725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:47:51.284 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:47:51.284833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:01.375 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:01.376674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:01.384 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:01.385658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:11.524 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:11.524807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:11.536 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:11.537689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:21.639 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:21.640255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:21.648 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:21.649167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:31.794 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:31.794789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:31.806 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:31.807637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:41.904 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:41.905168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:41.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:41.927900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:51.988 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:51.989058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:48:52.004 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:48:52.003974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:02.083 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:02.083496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:02.097 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:02.097983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:12.196 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:12.197008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:12.212 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:12.212475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:22.317 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:22.317994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:22.328 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:22.329349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:32.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:32.449475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:32.463 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:32.463313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:42.583 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:42.583025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:42.597 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:42.597964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:52.730 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:52.730356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:49:52.742 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:49:52.742958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:02.851 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:02.851776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:02.861 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:02.862370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:13.017 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:13.017002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:13.027 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:13.027928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:23.242 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:23.242367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:23.252 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:23.253026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:33.415 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:33.415202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:33.428 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:33.428870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:43.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:43.535682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:43.549 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:43.549522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:50.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:50.217249 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T21:50:50.869 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:50.869520 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T21:50:50.856+0000 7f0a5f6d1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T21:50:50.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:50:50.856+0000 7f0a5f6d1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T21:50:50.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T21:50:50.856+0000 7f0a5f6d1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T21:50:50.870 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T21:50:53.680 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:53.680289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:50:53.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:50:53.691901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:03.769 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:03.768758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:03.781 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:03.781516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:13.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:13.835628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:13.849 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:13.849249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:23.913 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:23.913413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:23.982 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:23.981832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:34.051 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:34.050811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:34.063 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:34.063297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:44.194 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:44.193807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:44.323 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:44.206602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:54.281 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:54.280877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:51:54.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:51:54.293499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:04.402 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:04.401951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:04.414 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:04.414672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:14.487 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:14.486893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:14.499 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:14.499167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:24.613 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:24.613355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:24.626 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:24.626782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:34.736 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:34.736346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:34.748 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:34.748275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:44.866 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:44.866316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:44.880 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:44.880610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:54.953 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:54.952518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:52:54.964 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:52:54.964127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:05.081 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:05.081020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:05.111 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:05.111044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:15.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:15.212680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:15.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:15.226068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:25.351 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:25.351120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:25.366 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:25.364038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:35.471 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:35.471011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:35.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:35.485808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:45.593 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:45.592743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:45.606 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:45.605916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:55.705 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:55.705141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:53:55.715 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:53:55.715166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:05.872 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:05.871657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:05.888 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:05.888108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:16.064 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:16.063030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:16.078 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:16.078130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:26.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:26.212660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:26.225 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:26.224320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:36.296 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:36.295378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:36.306 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:36.305654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:46.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:46.377725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:46.390 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:46.389635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:56.543 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:56.542962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:54:56.553 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:54:56.553097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:06.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:06.634080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:06.648 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:06.647992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:16.753 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:16.752249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:16.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:16.785376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:26.864 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:26.863087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:26.874 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:26.873777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:36.949 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:36.948986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:36.958 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:36.957808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:47.028 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:47.027675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:47.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:47.036384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:57.142 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:57.141229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:55:57.214 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:55:57.212936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:07.277 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:07.276339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:07.292 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:07.291088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:17.392 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:17.390755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:17.454 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:17.453698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:27.528 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:27.527324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:27.541 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:27.540043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:37.631 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:37.630190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:37.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:37.645677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:47.763 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:47.762284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:47.781 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:47.780003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:57.863 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:57.862119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:56:57.876 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:56:57.875642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:07.991 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:07.989705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:08.003 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:08.002943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:18.098 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:18.096788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:18.153 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:18.152292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:28.216 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:28.215067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:28.229 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:28.228609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:38.298 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:38.297092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:38.311 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:38.310496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:48.449 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:48.448127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:48.462 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:48.461054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:58.583 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:58.581635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:57:58.594 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:57:58.593271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:08.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:08.718411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:08.729 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:08.728474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:18.817 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:18.815529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:18.829 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:18.828138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:28.938 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:28.937085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:28.951 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:28.950219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:39.020 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:39.019302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:39.034 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:39.033061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:49.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:49.134703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:49.148 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:49.147340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:59.218 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:59.216548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:58:59.230 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:58:59.229057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:09.344 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:09.342640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:09.356 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:09.354655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:19.454 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:19.452871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:19.466 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:19.464963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:29.533 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:29.531613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:29.575 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:29.574133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:39.669 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:39.667855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:39.681 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:39.680030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:49.779 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:49.777706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:49.792 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:49.790130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:59.866 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:59.865117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:59:59.878 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 21:59:59.877214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:10.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:10.035489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:10.049 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:10.048136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:20.159 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:20.156590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:20.170 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:20.168581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:30.265 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:30.263940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:30.278 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:30.277385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:40.389 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:40.387494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:40.404 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:40.402976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:50.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:50.490119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:00:50.504 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:00:50.503148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:00.580 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:00.578380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:00.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:00.587728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:10.655 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:10.653610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:10.666 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:10.665002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:20.738 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:20.735933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:20.749 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:20.747843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:30.825 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:30.823108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:30.838 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:30.836335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:40.909 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:40.907139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:40.921 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:40.920115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:51.015 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:51.013601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:01:51.029 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:01:51.027277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:01.126 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:01.124438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:01.139 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:01.137760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:11.253 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:11.251892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:11.268 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:11.266096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:21.334 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:21.331979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:21.346 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:21.344618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:31.415 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:31.413249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:31.428 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:31.426860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:41.530 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:41.528579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:41.542 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:41.540444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:51.647 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:51.644719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:02:51.661 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:02:51.658323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:01.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:01.759213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:01.776 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:01.774573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:11.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:11.895359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:11.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:11.908618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:21.993 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:21.991662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:22.002 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:22.000689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:32.080 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:32.078300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:32.155 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:32.153407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:42.307 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:42.304908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:42.383 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:42.381241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:52.662 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:52.660396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:03:52.687 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:03:52.685139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:02.819 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:02.815580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:02.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:02.829330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:12.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:12.909829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:12.923 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:12.921276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:23.041 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:23.038695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:23.056 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:23.054016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:33.204 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:33.202126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:33.216 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:33.214069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:43.284 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:43.281743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:43.299 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:43.296498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:53.360 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:53.358284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:04:53.371 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:04:53.369047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:03.472 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:03.470213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:03.485 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:03.482786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:13.606 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:13.603501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:13.617 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:13.615555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:23.712 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:23.710019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:23.725 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:23.722687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:33.815 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:33.812225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:33.828 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:33.826090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:43.897 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:43.894575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:43.927 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:43.925351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:54.000 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:53.998208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:05:54.010 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:05:54.008207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:04.087 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:04.084289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:04.136 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:04.134086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:14.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:14.215141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:14.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:14.226238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:24.298 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:24.295685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:24.338 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:24.335380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:34.406 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:34.402819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:34.417 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:34.414540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:44.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:44.472467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:44.490 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:44.487858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:54.566 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:54.563920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:06:54.579 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:06:54.576457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:04.639 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:04.637017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:04.648 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:04.646080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:14.744 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:14.741885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:14.758 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:14.755631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:24.870 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:24.867815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:24.884 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:24.881585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:30.898 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:30.895212 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T22:07:31.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:31.537021 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T22:07:31.525+0000 7ff37ac6c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T22:07:31.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T22:07:31.525+0000 7ff37ac6c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T22:07:31.540 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T22:07:31.525+0000 7ff37ac6c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T22:07:31.541 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T22:07:34.986 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:34.983687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:34.999 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:34.996621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:45.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:45.076722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:45.094 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:45.091699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:55.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:55.187777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:07:55.204 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:07:55.202264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:05.331 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:05.328374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:05.460 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:05.341105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:15.414 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:15.411790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:15.426 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:15.423167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:25.535 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:25.532105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:25.543 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:25.540597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:35.678 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:35.674913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:35.690 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:35.687519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:45.753 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:45.750957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:45.768 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:45.766272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:55.851 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:55.848080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:08:55.863 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:08:55.860531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:06.007 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:06.004557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:06.048 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:06.045877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:16.258 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:16.255444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:16.270 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:16.267665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:26.391 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:26.388250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:26.403 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:26.400384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:36.597 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:36.593943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:36.610 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:36.607827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:46.714 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:46.710771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:46.727 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:46.724577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:56.789 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:56.785933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:09:56.800 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:09:56.797623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:06.898 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:06.895374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:06.910 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:06.907784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:16.982 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:16.976345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:16.992 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:16.989073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:27.106 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:27.103242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:27.127 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:27.122959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:37.226 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:37.223128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:37.240 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:37.237265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:47.301 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:47.298202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:47.353 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:47.350747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:57.419 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:57.415484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:10:57.432 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:10:57.428912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:07.494 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:07.491092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:07.508 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:07.505377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:17.638 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:17.634505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:17.651 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:17.648465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:27.722 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:27.719001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:27.736 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:27.733191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:37.819 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:37.815730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:37.874 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:37.871066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:47.944 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:47.941057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:47.958 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:47.954699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:58.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:58.073873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:11:58.089 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:11:58.086360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:08.154 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:08.151158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:08.168 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:08.164731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:18.279 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:18.275763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:18.293 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:18.290054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:28.483 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:28.480043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:28.493 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:28.490539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:38.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:38.566447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:38.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:38.641795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:48.750 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:48.746679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:48.765 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:48.762391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:58.892 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:58.889208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:12:58.906 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:12:58.902791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:08.969 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:08.965420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:08.980 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:08.976779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:19.090 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:19.086856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:19.104 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:19.101011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:29.213 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:29.209564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:29.225 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:29.221823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:39.340 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:39.336506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:39.355 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:39.351366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:49.458 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:49.455141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:49.470 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:49.466571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:59.578 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:59.573806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:13:59.586 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:13:59.583238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:09.691 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:09.687472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:09.707 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:09.703775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:19.809 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:19.805641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:19.821 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:19.817709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:29.917 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:29.913334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:29.929 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:29.926305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:40.005 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:40.001387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:40.013 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:40.009957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:50.111 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:50.108188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:14:50.126 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:14:50.122424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:00.238 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:00.234618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:00.249 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:00.246332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:10.418 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:10.414379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:10.430 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:10.426702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:20.563 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:20.559369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:20.574 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:20.571055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:30.704 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:30.700495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:30.719 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:30.714752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:40.863 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:40.859549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:40.877 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:40.873312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:50.982 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:50.978538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:15:50.995 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:15:50.991313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:01.124 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:01.120786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:01.140 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:01.136503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:11.220 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:11.215986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:11.234 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:11.230610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:21.341 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:21.337464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:21.355 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:21.351955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:31.488 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:31.484860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:31.503 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:31.499341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:41.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:41.573247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:41.592 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:41.588507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:51.697 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:51.693873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:16:51.710 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:16:51.706604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:01.811 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:01.807875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:01.824 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:01.820769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:11.941 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:11.937600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:11.954 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:11.950865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:22.061 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:22.058004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:22.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:22.071877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:32.174 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:32.170848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:32.188 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:32.185198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:42.327 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:42.324156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:42.345 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:42.342441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:52.416 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:52.412989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:17:52.429 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:17:52.425875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:02.520 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:02.517482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:02.532 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:02.529129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:12.629 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:12.625992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:12.646 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:12.643423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:22.786 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:22.783382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:22.796 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:22.793429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:32.898 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:32.895751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:32.912 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:32.909284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:43.035 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:43.031568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:43.045 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:43.042308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:53.177 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:53.174131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:18:53.189 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:18:53.187128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:03.283 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:03.280211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:03.294 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:03.291595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:13.361 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:13.358642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:13.380 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:13.373762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:23.501 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:23.498948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:23.515 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:23.512572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:33.589 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:33.586584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:33.601 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:33.598976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:43.717 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:43.714155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:43.730 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:43.727848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:53.831 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:53.828114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:19:53.843 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:19:53.841247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:03.913 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:03.909808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:03.959 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:03.957317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:14.037 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:14.035125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:14.052 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:14.049140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:24.114 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:24.111060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:24.126 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:24.123660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:34.197 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:34.194859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:34.210 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:34.208375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:44.319 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:44.315829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:44.329 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:44.327697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:54.437 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:54.434780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:20:54.450 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:20:54.447941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:04.519 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:04.516933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:04.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:04.574301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:14.717 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:14.714851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:14.730 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:14.728096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:24.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:24.795680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:24.807 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:24.805200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:34.875 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:34.872592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:34.888 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:34.886084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:44.982 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:44.980129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:44.994 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:44.992541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:55.131 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:55.129333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:21:55.145 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:21:55.143087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:05.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:05.215097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:05.228 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:05.227023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:15.296 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:15.293921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:15.315 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:15.313282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:25.385 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:25.383467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:25.395 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:25.394199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:35.475 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:35.473363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:35.486 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:35.484478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:45.555 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:45.553594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:45.569 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:45.567295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:55.635 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:55.633108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:22:55.649 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:22:55.647682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:05.749 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:05.747141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:05.761 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:05.759929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:15.977 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:15.975647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:15.987 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:15.985946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:26.151 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:26.149970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:26.163 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:26.161263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:36.288 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:36.286836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:36.301 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:36.299927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:46.373 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:46.371573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:46.385 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:46.384311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:56.479 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:56.477770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:23:56.492 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:23:56.490871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:06.564 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:06.562856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:06.577 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:06.575744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:11.546 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:11.544945 I | op-mon: parsing mon endpoints: c=10.105.216.166:6789,a=10.105.182.125:6789,b=10.104.249.171:6789 2022-02-21T22:24:12.272 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:12.270326 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T22:24:12.261+0000 7f3e9a75c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T22:24:12.272 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T22:24:12.261+0000 7f3e9a75c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T22:24:12.273 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21T22:24:12.261+0000 7f3e9a75c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T22:24:12.273 INFO:tasks.rook.operator.smithi081.stdout:couldn't init storage provider. : exit status 5 2022-02-21T22:24:16.700 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:16.698801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:16.716 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:16.715116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:26.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:26.833314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:26.850 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:26.848753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:36.953 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:36.951609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:36.969 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:36.967857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:47.074 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:47.072941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:47.086 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:47.084578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:57.194 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:57.192053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:24:57.206 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:24:57.204933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:07.282 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:07.281359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:07.296 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:07.294868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:17.367 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:17.365853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:17.378 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:17.377376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:27.478 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:27.476919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:27.493 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:27.491900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:37.570 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:37.568628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:37.581 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:37.580005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:47.700 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:47.699138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:47.714 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:47.712811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:57.785 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:57.783597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:25:57.797 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:25:57.795825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:07.891 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:07.890327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:07.903 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:07.902604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:18.065 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:18.063786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:18.075 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:18.073732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:28.171 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:28.169887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:28.190 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:28.184545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:38.283 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:38.281206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:38.298 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:38.297051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:48.397 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:48.394684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:48.409 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:48.407671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:58.522 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:58.520778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:26:58.534 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:26:58.533106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:08.613 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:08.612101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:08.648 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:08.647507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:18.725 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:18.724081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:18.739 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:18.738023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:28.835 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:28.833601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:28.848 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:28.847371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:38.965 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:38.963778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:38.985 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:38.983236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:49.066 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:49.064491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:49.079 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:49.078189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:59.185 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:59.184096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:27:59.199 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:27:59.197898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:09.301 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:09.300214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:09.314 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:09.312018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:19.405 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:19.403728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:19.419 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:19.418245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:29.484 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:29.483375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:29.543 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:29.542394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:39.628 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:39.626581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:39.641 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:39.640116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:49.723 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:49.722076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:49.732 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:49.731310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:59.889 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:59.887750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:28:59.898 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:28:59.897183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:09.957 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:09.955823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:09.970 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:09.965202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:20.077 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:20.075715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:20.100 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:20.098844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:30.161 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:30.160668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:30.217 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:30.216027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:40.289 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:40.288041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:40.347 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:40.346047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:50.413 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:50.412122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:29:50.424 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:29:50.423162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:30:00.505 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:30:00.504865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:30:00.519 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:30:00.518470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:30:10.588 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:30:10.587811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T22:30:10.599 INFO:tasks.rook.operator.smithi081.stdout:2022-02-21 22:30:10.598657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-22T00:50:50.828 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-02-22T00:50:50.851 DEBUG:teuthology.task.console_log:Killing console logger for smithi081 2022-02-22T00:50:50.853 DEBUG:teuthology.task.console_log:Killing console logger for smithi140 2022-02-22T00:50:50.853 DEBUG:teuthology.task.console_log:Killing console logger for smithi179 2022-02-22T00:50:50.853 DEBUG:teuthology.exit:Finished running handlers