2022-03-08T22:45:02.117 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:45:02.119 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283 --owner scheduled_yuriw@teuthology --name yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi --verbose --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} -- /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283/orig.config.yaml 2022-03-08T22:45:02.154 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:45:02.217 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283 branch: wip-yuri-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '6727283' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri-testing-2022-03-07-0958 sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 targets: smithi042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp+r3NZ9xk/LPbYhen5HoJVg+3CYPyM0ifBHLaHNDu8ivA4Uh6oOHwNTdCk6c8uwweuJJJeC5qV61tieTnFU/H4WcSQ9n6Grq817UpJ9DKO8XSeBnEVM4uvzs24h0rMjw/y0Du++qdjcYqsHCUgy5FfOKSaAlbhuTzMCeHArgFEbcHreUV8tLuvYrxg4Pfc8mZLtRihuj/DM42DNpewK1Mmqb574SGVhSP/tDAS/lJOtIXSSl5alLRZ9bp7cFuLAiFBZfSgLD+UGOTX/hHf3DTKha5mCfsvOijbx2ffEDYapWk7fSPcZElxn32WuwSsQ9fm+A/WOa5YdUpBtKtePhCqiEJS/8ihGIP6WTtj9OwLv4Hvd1h9DZUZ8o8N0+/phx5j1m0GKNLE8aeyFn9WprmruxpJRtRKcCjulKpCAODLq9jeYapuLV1W+rx883z67khDVEMaLt9ELTmLjHD4ok1egzpCmm+A5HQg7M9dKQXwkC2yVXvA8Kpy/mpRIKQOH8= smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSArKGRVrrX6hBKsSann1nrjp9i/tDtIGgHfDVSFcWnqZvp+Ok/7nwzFSFA3ADhbvm8SGDnPQYTl37zP7vuq16rDwVb+Sw7xXrKYFgRI+pqhFH5qP4Y+U9NRyVcdY8DnMzKFYOLugPbzjDi6Z/v43glDhzxl4gngFhGcLUmbBfFCdomqs4s1W5tAT69ScRAuZ/xkDou4TGXTjB+fdXYK8oEOK+I7i/v4jQD4nmbtClVru33aRuiaGy9ZD/qqIePIAkc4o6favbVZqLrtzd9sYK8hLEumeaNp+oajnjSPSPEcMOijeHDLH/u3P50CvD5tM1wpV/rmyyfVCEuU+alTQvZJ7EuHDSjODBGIWEHPojciGzcS3ZRg7KjJYHMJ0PrlpDGAtIUhcf/UIXuPJ8qbhGu1JmPuKIv6l8cUNrj2/rARuYiBZa772L4bWnngNHXN2ZdWIhrSxSZjZks24uI7tLfGWjl/biez3GXL8m6tMK7P/qTl0Bd2B2V1fWb+44Dzk= smithi178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpnLUN9vxdEdYVnZKGbn/FCL9ZrfNNncWuY75y/tw2iguYjxJvvizN1QYVQEIlekf5Lx09idUkEW4v6fp1Vm/HC0Y6jBz/JMr4pCZvQfOCynnL3YJoF1f38h1UZ8XVcyRScIsibd7x/1Fk30lho1w854xqxTHCtJ2d8btZBfiVvPJ1/Dqsu8MjpLVQicY6sM+5gRt1ZuurTntq85+CabTIJCwXzP5TnA6N9ZpQOLPcZAJceeUniTN5c8E+AmnketWrXFBNwWMi5lG0vWMZQXMDP8PxsXMXn2XzIqV9mtJTGID4Naz3Be138K0Exr0YN2GKBPsMhBBvHep4rgF42AO6sTq4VrDrr4neUqTQNg/4SF0ETIv90se5wzW7L11QIHkbtFoqq22d33oKxS5Sg2JGwVueRpok+c9a+FEXcf46xLITqff3s4dGDXwakDa8gCvRkCxoCUnBw6hgpatLIbv9k8EA7sOca0I1K4HEVBoPwlCnlAlauh80+ebietjHTq8= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:35:06 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:45:02.218 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa; will attempt to use it 2022-03-08T22:45:02.219 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks 2022-03-08T22:45:02.219 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:45:02.221 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:45:02.244 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '94cb3430b74fb099b8927e3e6ad50dfd969173f4' 2022-03-08T22:45:02.244 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:45:02.245 INFO:teuthology.packaging:ref: None 2022-03-08T22:45:02.245 INFO:teuthology.packaging:tag: None 2022-03-08T22:45:02.245 INFO:teuthology.packaging:branch: wip-yuri-testing-2022-03-07-0958 2022-03-08T22:45:02.246 INFO:teuthology.packaging:sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:45:02.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-testing-2022-03-07-0958 2022-03-08T22:45:02.402 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:45:02.403 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:45:02.415 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:45:02.415 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:45:02.463 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:45:02.476 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:45:02.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:45:02.521 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:39:20.608920', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK6N+w6n+1C7v8LvsHkRVUsrbq7joQfpfvzQgfN3YyT'} 2022-03-08T22:45:02.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:39:20.607967', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFmtPWoC2Al7GM761m6bwbj/wHrrkV/Fp0lKr4lDYTg/'} 2022-03-08T22:45:02.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:39:20.606512', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINUQoW4QSZmjeS6lSca43rkMEX/xHZkMpdVzuDjH0f6X'} 2022-03-08T22:45:02.565 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:45:02.576 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:45:02.577 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-08T22:45:02.577 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-08T22:45:02.577 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:45:02.702 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fab6eddd9d8>, signals=[15]) 2022-03-08T22:45:02.704 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:45:02.715 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:45:02.716 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2022-03-08T22:45:02.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:45:02.772 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2022-03-08T22:45:02.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:45:02.824 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2022-03-08T22:45:02.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:45:02.878 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:45:02.890 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2022-03-08T22:45:02.898 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2022-03-08T22:45:02.898 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2022-03-08T22:45:02.948 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2022-03-08T22:45:02.948 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:45:02.948 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2022-03-08T22:45:02.949 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2022-03-08T22:45:02.949 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:45:02.949 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="20.04" 2022-03-08T22:45:02.949 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:45:02.950 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:45:02.950 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:45:02.950 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:45:02.950 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=focal 2022-03-08T22:45:02.951 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:45:02.951 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2022-03-08T22:45:02.970 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2022-03-08T22:45:02.977 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2022-03-08T22:45:02.977 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2022-03-08T22:45:03.025 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2022-03-08T22:45:03.026 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:45:03.026 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2022-03-08T22:45:03.026 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2022-03-08T22:45:03.026 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:45:03.027 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="20.04" 2022-03-08T22:45:03.027 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:45:03.027 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:45:03.027 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:45:03.028 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:45:03.028 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=focal 2022-03-08T22:45:03.028 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:45:03.029 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-03-08T22:45:03.045 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2022-03-08T22:45:03.052 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2022-03-08T22:45:03.052 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2022-03-08T22:45:03.101 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Ubuntu" 2022-03-08T22:45:03.102 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:45:03.102 INFO:teuthology.orchestra.run.smithi178.stdout:ID=ubuntu 2022-03-08T22:45:03.102 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE=debian 2022-03-08T22:45:03.102 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:45:03.103 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="20.04" 2022-03-08T22:45:03.103 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:45:03.103 INFO:teuthology.orchestra.run.smithi178.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:45:03.103 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:45:03.104 INFO:teuthology.orchestra.run.smithi178.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:45:03.104 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_CODENAME=focal 2022-03-08T22:45:03.104 INFO:teuthology.orchestra.run.smithi178.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:45:03.104 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2022-03-08T22:45:03.123 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:45:03.138 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:45:03.172 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:45:03.172 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:45:03.175 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:45:03.178 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:45:03.185 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:45:03.205 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:45:03.206 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:45:03.224 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:45:03.226 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:45:03.242 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:45:03.319 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:45:03.402 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:45:03.403 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:45:03.403 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-08T22:45:03.403 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:45:03.404 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2022-03-08T22:45:03.405 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2022-03-08T22:45:03.405 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2022-03-08T22:45:03.412 INFO:teuthology.orchestra.run.smithi153.stdout:5.4.0-97-generic 2022-03-08T22:45:03.412 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.4.0-97-generic 2022-03-08T22:45:03.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2022-03-08T22:45:03.415 INFO:teuthology.orchestra.run.smithi178.stdout:5.4.0-97-generic 2022-03-08T22:45:03.415 INFO:teuthology.orchestra.run.smithi042.stdout:5.4.0-97-generic 2022-03-08T22:45:03.415 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.4.0-97-generic 2022-03-08T22:45:03.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2022-03-08T22:45:03.416 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.4.0-97-generic 2022-03-08T22:45:03.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2022-03-08T22:45:03.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2022-03-08T22:45:03.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2022-03-08T22:45:03.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2022-03-08T22:45:03.727 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:45:03.769 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:45:03.769 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:45:03.822 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:45:03.822 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:45:03.833 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:45:03.837 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:45:03.869 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:45:03.930 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:45:04.059 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:45:04.179 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:45:04.204 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:45:04.237 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:45:04.246 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:45:04.251 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:45:04.251 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:45:04.252 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:45:04.277 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:45:04.280 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:45:04.280 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:45:04.284 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:45:04.295 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:45:04.311 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:45:04.312 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:45:04.313 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:45:04.323 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:45:04.324 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:45:04.325 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:45:04.325 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:04.333 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:45:04.415 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:45:04.432 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:45:04.468 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:45:04.584 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:45:04.638 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:45:04.704 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:45:04.753 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:45:04.757 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:45:04.772 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:45:04.773 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:45:04.800 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:45:04.815 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:45:04.822 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:45:04.822 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:45:04.823 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:45:04.830 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:45:04.831 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:45:04.831 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:45:04.836 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:45:04.837 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:45:04.845 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:45:04.845 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:45:04.859 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:45:04.880 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:45:04.886 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:45:04.890 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:45:04.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:45:04.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:45:04.896 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:45:04.897 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:45:04.899 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:45:04.900 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:45:04.900 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:04.900 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:45:04.907 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:45:04.907 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:45:04.922 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:45:04.925 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:45:04.941 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:45:04.942 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:45:04.942 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:45:04.946 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:04.947 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:45:04.948 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:45:04.952 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:45:04.953 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:45:04.953 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:45:04.954 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:45:04.955 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:45:04.955 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:45:04.956 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:04.960 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:45:04.966 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:45:04.977 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:45:04.988 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:45:04.992 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:45:04.993 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:45:04.993 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:04.994 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:45:04.995 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:45:04.995 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:45:04.995 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:45:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:45:05.003 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:45:05.005 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:45:05.015 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:45:05.019 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:45:05.020 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:45:05.020 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:45:05.021 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:45:05.033 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:45:05.033 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:45:05.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:45:05.035 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:45:05.035 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:45:05.047 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:45:05.051 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:45:05.061 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:45:05.062 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:45:05.076 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:45:05.077 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:45:05.077 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:45:05.078 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:45:05.079 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:45:05.087 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:45:05.088 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:45:05.088 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:45:05.089 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:45:05.090 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:45:05.095 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:45:05.143 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:45:05.144 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:45:05.144 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:45:05.144 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:45:05.177 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:45:05.178 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:45:05.179 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:45:05.180 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:45:05.180 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:45:07.638 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 10.2 MB in 2s (5,608 kB/s) 2022-03-08T22:45:07.855 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 10.2 MB in 2s (4,495 kB/s) 2022-03-08T22:45:07.858 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 10.2 MB in 2s (4,555 kB/s) 2022-03-08T22:45:08.843 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:45:08.867 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:45:08.939 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:45:09.015 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:45:09.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:45:09.063 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:45:09.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:45:09.105 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:45:09.118 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:45:09.118 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:45:09.154 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:45:09.277 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2022-03-08T22:45:09.278 INFO:teuthology.orchestra.run.smithi042.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.278 INFO:teuthology.orchestra.run.smithi042.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:45:09.279 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:45:09.279 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2022-03-08T22:45:09.280 INFO:teuthology.orchestra.run.smithi042.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:45:09.280 INFO:teuthology.orchestra.run.smithi042.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:45:09.283 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:45:09.284 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:45:09.310 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:45:09.311 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:45:09.331 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2022-03-08T22:45:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:45:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:45:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: thermald upower usbmuxd 2022-03-08T22:45:09.436 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2022-03-08T22:45:09.436 INFO:teuthology.orchestra.run.smithi153.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.436 INFO:teuthology.orchestra.run.smithi153.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:45:09.437 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:45:09.437 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2022-03-08T22:45:09.437 INFO:teuthology.orchestra.run.smithi153.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:45:09.438 INFO:teuthology.orchestra.run.smithi153.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:45:09.443 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2022-03-08T22:45:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:45:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:45:09.444 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2022-03-08T22:45:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:45:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:45:09.487 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2022-03-08T22:45:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:45:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:45:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: thermald upower usbmuxd 2022-03-08T22:45:09.493 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2022-03-08T22:45:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:45:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:45:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:45:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: thermald upower usbmuxd 2022-03-08T22:45:09.518 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:45:09.518 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:45:09.518 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:45:09.518 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:45:09.669 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:45:09.669 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:45:09.670 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:45:09.670 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:45:09.678 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:45:09.678 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:45:09.678 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:45:09.679 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:45:09.780 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:45:09.794 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:45:09.795 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:45:09.882 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:45:09.883 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:45:09.919 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:45:09.920 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:45:09.928 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:45:09.947 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:45:09.951 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:45:09.964 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:45:10.021 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:45:10.023 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:45:10.039 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:45:10.055 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:45:11.084 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:45:11.361 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:45:11.370 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:45:11.419 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:45:11.676 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:45:11.698 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:45:13.335 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:45:13.335 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:45:13.336 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:45:13.341 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:45:13.343 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:45:13.568 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:45:13.569 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:45:13.569 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:45:13.576 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:45:13.577 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:45:13.672 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:45:13.673 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:45:13.673 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:45:13.679 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:45:13.681 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:45:13.775 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 80.1 MB in 4s (20.0 MB/s) 2022-03-08T22:45:13.943 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:45:13.973 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 80.1 MB in 4s (19.6 MB/s) 2022-03-08T22:45:14.096 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 80.1 MB in 4s (19.1 MB/s) 2022-03-08T22:45:14.174 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:45:14.305 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:45:15.384 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:45:15.389 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:45:15.514 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:15.696 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:45:15.700 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:45:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:45:15.791 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:45:15.837 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:15.892 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:45:15.905 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:45:15.910 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:15.942 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:16.182 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:45:16.195 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:45:16.232 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:16.263 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:45:16.270 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:45:16.276 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:45:16.283 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:45:16.321 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:16.345 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:16.543 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:45:16.558 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:45:16.601 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:16.641 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:45:16.657 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:45:16.690 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:45:16.699 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:16.703 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:45:16.740 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:16.954 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:45:16.968 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:45:17.003 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:45:17.012 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:17.016 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:45:17.061 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:17.176 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:45:17.192 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:17.234 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:17.365 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:45:17.379 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:17.405 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:45:17.420 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:17.432 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:17.464 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:17.554 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:45:17.567 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:17.629 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:17.701 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:45:17.717 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:17.760 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:17.791 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:45:17.806 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:17.850 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:19.846 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:45:19.860 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:19.964 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:20.009 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:45:20.021 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:20.026 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:45:20.039 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:20.113 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:20.141 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:20.620 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:45:20.633 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:45:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:20.646 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:45:20.648 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:20.659 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:45:20.678 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:20.682 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:20.704 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:26.558 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:45:26.574 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:45:26.597 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:45:26.616 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:26.617 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:45:26.645 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:45:26.651 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:26.663 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:45:26.704 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:26.961 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:45:26.975 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:45:26.978 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:45:26.992 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:45:27.016 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:45:27.018 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:27.031 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:45:27.037 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:27.074 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:27.365 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package thermald. 2022-03-08T22:45:27.380 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package thermald. 2022-03-08T22:45:27.381 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:45:27.395 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:45:27.414 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:27.419 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package thermald. 2022-03-08T22:45:27.432 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:45:27.461 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:27.463 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:27.972 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package upower. 2022-03-08T22:45:27.985 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:28.005 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package upower. 2022-03-08T22:45:28.018 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package upower. 2022-03-08T22:45:28.019 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:28.032 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:45:28.032 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:45:28.095 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:45:28.098 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:45:28.439 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:45:28.452 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:45:28.460 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:45:28.475 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:45:28.481 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:45:28.495 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:45:28.497 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:28.531 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:28.577 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:28.802 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:28.831 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:28.865 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:45:28.926 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:28.952 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:28.982 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:45:29.052 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:29.078 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:29.091 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:45:29.178 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:29.205 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:29.233 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:45:29.304 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:29.356 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:29.359 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:45:29.430 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:29.482 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:29.485 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:45:29.565 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:45:29.608 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:45:29.611 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:45:29.974 INFO:teuthology.orchestra.run.smithi153.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:30.016 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:30.058 INFO:teuthology.orchestra.run.smithi042.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:30.071 INFO:teuthology.orchestra.run.smithi178.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:30.101 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:30.130 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:45:30.190 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:45:30.199 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:45:30.200 INFO:teuthology.orchestra.run.smithi153.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:45:30.282 INFO:teuthology.orchestra.run.smithi042.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:45:30.292 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:45:30.293 INFO:teuthology.orchestra.run.smithi042.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:45:30.309 INFO:teuthology.orchestra.run.smithi178.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:45:30.317 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:45:30.317 INFO:teuthology.orchestra.run.smithi178.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:45:30.683 INFO:teuthology.orchestra.run.smithi153.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:45:30.821 INFO:teuthology.orchestra.run.smithi178.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:45:30.899 INFO:teuthology.orchestra.run.smithi153.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:30.909 INFO:teuthology.orchestra.run.smithi042.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:45:30.938 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:31.050 INFO:teuthology.orchestra.run.smithi178.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:31.093 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:31.152 INFO:teuthology.orchestra.run.smithi042.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:31.206 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:31.402 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.403 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.596 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.596 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.716 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.716 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:31.880 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:32.175 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:32.187 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:33.123 INFO:teuthology.orchestra.run.smithi178.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:45:33.123 INFO:teuthology.orchestra.run.smithi178.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:45:33.299 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:33.431 INFO:teuthology.orchestra.run.smithi153.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:45:33.431 INFO:teuthology.orchestra.run.smithi153.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:45:33.490 INFO:teuthology.orchestra.run.smithi042.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:45:33.490 INFO:teuthology.orchestra.run.smithi042.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:45:33.610 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:33.648 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:34.634 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:34.945 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:35.008 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:35.943 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:36.070 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:45:36.271 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:36.388 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:45:36.635 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:36.761 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:45:40.592 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:45:40.710 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:45:40.941 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:45:41.277 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:45:41.378 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:41.379 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:45:41.471 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:41.584 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:45:41.654 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:45:41.654 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:41.685 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:41.876 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:45:41.876 INFO:teuthology.orchestra.run.smithi178.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:42.191 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:45:42.192 INFO:teuthology.orchestra.run.smithi042.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:04.874 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:46:04.875 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:46:04.882 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:04.902 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2022-03-08T22:46:05.200 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:05.227 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:05.336 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:46:05.336 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:46:05.349 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:05.373 INFO:teuthology.orchestra.run.smithi178.stdout:Generating grub configuration file ... 2022-03-08T22:46:05.603 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:05.614 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:05.729 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:06.115 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:46:06.115 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:46:06.130 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:06.159 INFO:teuthology.orchestra.run.smithi042.stdout:Generating grub configuration file ... 2022-03-08T22:46:06.186 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:06.198 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:06.448 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:06.481 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:06.835 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:06.846 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:07.957 INFO:teuthology.orchestra.run.smithi178.stdout:done 2022-03-08T22:46:08.292 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:08.295 INFO:teuthology.orchestra.run.smithi153.stdout:done 2022-03-08T22:46:08.324 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:08.324 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2022-03-08T22:46:08.324 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2022-03-08T22:46:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2022-03-08T22:46:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 19 2022-03-08T22:46:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2022-03-08T22:46:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2022-03-08T22:46:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:08.327 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:08.327 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2022-03-08T22:46:08.328 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:46:08.328 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:46:08.328 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:46:08.329 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-08T22:46:08.329 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:46:08.329 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2022-03-08T22:46:08.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2022-03-08T22:46:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:46:08.631 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:08.661 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:08.661 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2022-03-08T22:46:08.661 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2022-03-08T22:46:08.662 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2022-03-08T22:46:08.662 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 19 2022-03-08T22:46:08.662 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:08.662 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2022-03-08T22:46:08.663 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:08.663 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:08.663 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:08.663 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:08.664 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2022-03-08T22:46:08.664 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2022-03-08T22:46:08.665 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:46:08.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:46:08.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:46:08.665 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-08T22:46:08.666 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:46:08.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2022-03-08T22:46:08.667 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:46:08.744 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:46:08.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2022-03-08T22:46:08.827 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:46:09.030 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:46:09.031 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:46:09.103 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:46:09.189 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:46:09.459 INFO:teuthology.orchestra.run.smithi042.stdout:done 2022-03-08T22:46:09.848 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:09.880 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:09.987 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2022-03-08T22:46:09.987 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2022-03-08T22:46:09.988 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2022-03-08T22:46:09.988 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 19 2022-03-08T22:46:09.988 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:09.988 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2022-03-08T22:46:09.989 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:09.989 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:09.989 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:09.989 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:09.990 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2022-03-08T22:46:09.990 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:09.990 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:09.990 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2022-03-08T22:46:09.991 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:46:09.991 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:46:09.991 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:46:09.991 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:46:09.992 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:46:09.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2022-03-08T22:46:10.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2022-03-08T22:46:10.086 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:46:10.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:46:10.177 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:46:10.195 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:46:10.308 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:46:10.332 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:46:10.332 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:46:10.394 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:46:10.489 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:46:10.502 INFO:teuthology.orchestra.run.smithi178.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:46:10.503 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:46:10.505 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:10.526 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:10.526 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2022-03-08T22:46:10.526 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2022-03-08T22:46:10.527 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2022-03-08T22:46:10.527 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 19 2022-03-08T22:46:10.527 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:10.527 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2022-03-08T22:46:10.528 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:10.528 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:10.528 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:10.528 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:10.528 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2022-03-08T22:46:10.529 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:10.529 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:10.530 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2022-03-08T22:46:10.530 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2022-03-08T22:46:10.537 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.WNL4cXOg5i 2022-03-08T22:46:10.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WNL4cXOg5i 2022-03-08T22:46:10.570 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:46:10.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.WNL4cXOg5i 2022-03-08T22:46:10.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:46:10.638 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.WNL4cXOg5i is 8KB 2022-03-08T22:46:10.652 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.WNL4cXOg5i 2022-03-08T22:46:10.659 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:10.660 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:46:10.721 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:46:10.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:46:10.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:10.759 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:46:10.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo update-grub 2022-03-08T22:46:10.813 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:46:10.814 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:46:10.980 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:46:10.980 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:46:10.982 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:11.003 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:11.003 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2022-03-08T22:46:11.003 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2022-03-08T22:46:11.004 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2022-03-08T22:46:11.004 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 19 2022-03-08T22:46:11.004 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:11.004 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2022-03-08T22:46:11.005 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:11.005 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:11.005 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:11.005 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:11.005 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2022-03-08T22:46:11.006 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:11.006 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:11.006 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2022-03-08T22:46:11.006 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2022-03-08T22:46:11.052 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.tBh9jNuRMO 2022-03-08T22:46:11.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tBh9jNuRMO 2022-03-08T22:46:11.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.tBh9jNuRMO 2022-03-08T22:46:11.106 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.tBh9jNuRMO is 8KB 2022-03-08T22:46:11.118 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.tBh9jNuRMO 2022-03-08T22:46:11.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:11.179 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:46:11.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:46:11.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:11.208 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:46:11.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2022-03-08T22:46:11.782 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:46:11.804 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:46:11.871 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:46:11.886 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:46:11.895 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2022-03-08T22:46:12.049 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:46:12.050 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:46:12.178 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:12.190 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:12.243 INFO:teuthology.orchestra.run.smithi042.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:46:12.244 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:46:12.245 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:46:12.264 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:46:12.265 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2022-03-08T22:46:12.265 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2022-03-08T22:46:12.265 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2022-03-08T22:46:12.265 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 19 2022-03-08T22:46:12.265 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:46:12.266 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2022-03-08T22:46:12.266 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:46:12.266 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:46:12.266 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:46:12.267 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:46:12.267 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2022-03-08T22:46:12.267 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2022-03-08T22:46:12.267 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:46:12.268 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2022-03-08T22:46:12.268 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2022-03-08T22:46:12.272 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.pecUuKvxHe 2022-03-08T22:46:12.273 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub/grub.cfg /tmp/tmp.pecUuKvxHe 2022-03-08T22:46:12.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.pecUuKvxHe 2022-03-08T22:46:12.369 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:46:12.379 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:12.385 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.pecUuKvxHe is 8KB 2022-03-08T22:46:12.393 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2022-03-08T22:46:12.399 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.pecUuKvxHe 2022-03-08T22:46:12.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:12.463 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:46:12.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:46:12.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:46:12.495 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:46:12.496 DEBUG:teuthology.orchestra.run.smithi042:> sudo update-grub 2022-03-08T22:46:12.521 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:12.532 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:12.624 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:12.636 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:12.997 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:13.239 INFO:teuthology.orchestra.run.smithi178.stderr:done 2022-03-08T22:46:13.241 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2022-03-08T22:46:13.884 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:46:13.886 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:46:13.900 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2022-03-08T22:46:14.138 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:46:14.150 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:46:14.489 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:46:14.501 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:46:14.624 INFO:teuthology.orchestra.run.smithi153.stderr:done 2022-03-08T22:46:14.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2022-03-08T22:46:16.097 INFO:teuthology.orchestra.run.smithi042.stderr:done 2022-03-08T22:46:16.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2022-03-08T22:46:43.271 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-08T22:46:43.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2022-03-08T22:46:43.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:46:44.631 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-08T22:46:44.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-08T22:46:44.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:46:46.104 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:46:46.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2022-03-08T22:46:46.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:01.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2022-03-08T22:47:03.119 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2022-03-08T22:47:04.651 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2022-03-08T22:47:31.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2022-03-08T22:47:31.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:33.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-08T22:47:33.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:34.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2022-03-08T22:47:34.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:50.135 DEBUG:teuthology.orchestra.run.smithi042:> true 2022-03-08T22:47:50.219 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2022-03-08T22:47:50.656 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2022-03-08T22:47:50.657 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:47:50.657 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2022-03-08T22:47:50.664 INFO:teuthology.orchestra.run.smithi042.stdout:5.13.0-35-generic 2022-03-08T22:47:50.665 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:47:50.665 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:47:50.665 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:47:51.499 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2022-03-08T22:47:51.666 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:47:51.666 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:47:51.731 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2022-03-08T22:47:51.759 DEBUG:teuthology.parallel:result is None 2022-03-08T22:48:20.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2022-03-08T22:48:20.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:48:20.545 DEBUG:teuthology.orchestra.run.smithi178:> true 2022-03-08T22:48:21.075 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2022-03-08T22:48:21.075 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-08T22:48:21.076 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2022-03-08T22:48:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:5.13.0-35-generic 2022-03-08T22:48:21.124 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:48:21.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:48:21.125 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:48:21.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-08T22:48:21.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:48:21.689 DEBUG:teuthology.orchestra.run.smithi153:> true 2022-03-08T22:48:22.126 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-08T22:48:22.127 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:48:22.191 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2022-03-08T22:48:22.219 DEBUG:teuthology.parallel:result is None 2022-03-08T22:48:22.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-08T22:48:22.238 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-08T22:48:22.238 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2022-03-08T22:48:22.244 INFO:teuthology.orchestra.run.smithi153.stdout:5.13.0-35-generic 2022-03-08T22:48:22.244 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:48:22.244 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:48:22.244 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:48:23.245 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-08T22:48:23.246 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:48:23.308 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2022-03-08T22:48:23.335 DEBUG:teuthology.parallel:result is None 2022-03-08T22:48:23.336 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:48:23.344 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:48:23.345 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:48:23.348 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:48:23.350 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:48:23.358 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:48:23.398 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:48:23.431 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:48:23.432 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:48:23.435 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:48:23.437 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:48:23.538 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:48:23.547 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:48:23.547 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:48:23.550 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:48:23.553 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:48:23.575 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.576 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.578 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.583 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.585 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.586 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:48:23.588 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:48:23.598 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:48:23.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:48:23.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:48:23.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:48:23.619 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:48:23.631 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:48:23.632 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:48:23.634 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:48:23.663 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:48:23.670 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:48:23.690 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:48:23.696 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:48:23.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:48:23.718 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:48:23.726 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:48:23.776 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:48:23.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:48:23.794 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:48:23.802 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:48:23.852 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:48:23.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:48:23.899 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2022-03-08T22:48:23.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2022-03-08T22:48:23.905 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2022-03-08T22:48:23.951 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:48:23.962 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:48:23.962 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:48:24.003 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:48:24.039 DEBUG:teuthology.task.selinux:Excluding smithi042: OS 'ubuntu' does not support SELinux 2022-03-08T22:48:24.040 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2022-03-08T22:48:24.040 DEBUG:teuthology.task.selinux:Excluding smithi178: OS 'ubuntu' does not support SELinux 2022-03-08T22:48:24.040 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:48:24.041 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:48:24.041 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:48:24.041 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:48:24.071 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:48:24.071 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:48:24.102 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:48:24.104 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi153.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T22:51:30.798 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2022-03-08T22:51:30.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2022-03-08T22:51:30.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:30.849 DEBUG:teuthology.orchestra.run.smithi042:> true 2022-03-08T22:51:30.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-03-08T22:51:30.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:31.013 DEBUG:teuthology.orchestra.run.smithi153:> true 2022-03-08T22:51:31.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2022-03-08T22:51:31.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:31.191 DEBUG:teuthology.orchestra.run.smithi178:> true 2022-03-08T22:51:31.312 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T22:51:31.326 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T22:51:31.327 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:51:31.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:51:31.332 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:51:31.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:51:31.335 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:51:31.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:51:31.361 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:51:31.361 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Command line: ntpd -gq 2022-03-08T22:51:31.362 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: proto: precision = 0.082 usec (-23) 2022-03-08T22:51:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.363 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.365 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.365 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:51:31.365 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:51:31.365 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:51:31.366 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen normally on 3 ens1f0 172.21.15.42:123 2022-03-08T22:51:31.366 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen normally on 4 lo [::1]:123 2022-03-08T22:51:31.366 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1440%4]:123 2022-03-08T22:51:31.368 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:31 ntpd[8937]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:51:31.368 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:51:31.369 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Command line: ntpd -gq 2022-03-08T22:51:31.369 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: proto: precision = 0.100 usec (-23) 2022-03-08T22:51:31.370 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.370 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:51:31.370 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Command line: ntpd -gq 2022-03-08T22:51:31.370 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: proto: precision = 0.066 usec (-24) 2022-03-08T22:51:31.371 INFO:teuthology.orchestra.run.smithi178.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.371 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.372 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.372 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:51:31.373 INFO:teuthology.orchestra.run.smithi178.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.375 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.375 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:51:31.376 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:51:31.376 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:51:31.376 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2022-03-08T22:51:31.376 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 4 lo [::1]:123 2022-03-08T22:51:31.377 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2022-03-08T22:51:31.377 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:31 ntpd[9029]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:51:31.377 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:51:31.377 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:51:31.377 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:51:31.378 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:51:31.378 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 3 enp3s0f1 172.21.15.178:123 2022-03-08T22:51:31.378 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 4 lo [::1]:123 2022-03-08T22:51:31.378 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ef9%5]:123 2022-03-08T22:51:31.379 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:31 ntpd[9029]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:51:32.363 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:32 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.373 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.373 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:32.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:32.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:32.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:32.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:32 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.363 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:34 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:34.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:34.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:34.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:34.377 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:34 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:35.366 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.366 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:35.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:35.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.369 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:35.369 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:35.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:35.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:35.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:35.372 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:35 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.368 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.369 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.369 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.369 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.369 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.370 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:36 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.373 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.374 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.375 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.376 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.376 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.377 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.377 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.377 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.377 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.378 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.378 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.378 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.378 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.379 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.379 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.379 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.379 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.379 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:36.380 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:36.380 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:36.380 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:36.380 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:36 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:37.366 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.366 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.367 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:37.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:37.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.368 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:37.369 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:37.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:37.370 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:37.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:37.371 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:37 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:38.363 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.363 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.364 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.365 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:38.366 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: 8 Mar 22:51:38 ntpd[8937]: ntpd: time slew -0.008562 s 2022-03-08T22:51:38.366 INFO:teuthology.orchestra.run.smithi042.stdout:ntpd: time slew -0.008562s 2022-03-08T22:51:38.367 INFO:teuthology.orchestra.run.smithi042.stderr: 8 Mar 22:51:38 ntpd[8937]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:51:38.369 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.369 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.370 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.370 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.370 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.370 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.371 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:38.371 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:38.371 INFO:teuthology.orchestra.run.smithi178.stdout: 8 Mar 22:51:38 ntpd[9029]: ntpd: time slew -0.000292 s 2022-03-08T22:51:38.372 INFO:teuthology.orchestra.run.smithi178.stdout:ntpd: time slew -0.000292s 2022-03-08T22:51:38.372 INFO:teuthology.orchestra.run.smithi178.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:51:38.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.373 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.374 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.374 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.374 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.374 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.375 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:38.375 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:38.375 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:51:38.375 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:51:38.375 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:51:38.376 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:51:38.376 INFO:teuthology.orchestra.run.smithi153.stdout: 8 Mar 22:51:38 ntpd[9029]: ntpd: time slew -0.001390 s 2022-03-08T22:51:38.376 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew -0.001390s 2022-03-08T22:51:38.377 INFO:teuthology.orchestra.run.smithi153.stderr: 8 Mar 22:51:38 ntpd[9029]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:51:38.418 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:51:38.418 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2022-03-08T22:51:38.419 INFO:teuthology.orchestra.run.smithi042.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.419 INFO:teuthology.orchestra.run.smithi042.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.419 INFO:teuthology.orchestra.run.smithi042.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.419 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.430 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:51:38.431 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2022-03-08T22:51:38.431 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.431 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.432 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.432 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.480 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:51:38.480 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2022-03-08T22:51:38.480 INFO:teuthology.orchestra.run.smithi178.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.481 INFO:teuthology.orchestra.run.smithi178.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.481 INFO:teuthology.orchestra.run.smithi178.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.481 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:51:38.482 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T22:51:38.497 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T22:51:38.498 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-08T22:51:38.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:51:38.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:51:38.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:51:38.523 INFO:teuthology.orchestra.run.smithi042.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:51:38.529 INFO:teuthology.orchestra.run.smithi153.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:51:38.530 INFO:teuthology.orchestra.run.smithi042.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:51:38.531 INFO:teuthology.orchestra.run.smithi178.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:51:38.539 INFO:teuthology.orchestra.run.smithi153.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:51:38.540 INFO:teuthology.orchestra.run.smithi042.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:51:38.540 INFO:teuthology.orchestra.run.smithi178.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:51:38.545 INFO:teuthology.orchestra.run.smithi178.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:51:38.549 INFO:teuthology.orchestra.run.smithi153.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:51:38.612 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:51:38.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:51:38.646 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:51:38.647 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:51:38.657 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:51:38.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:51:38.673 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:51:38.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:51:38.683 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:51:38.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:51:38.699 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:51:38.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:51:38.713 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl restart docker || true 2022-03-08T22:51:38.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl restart docker || true 2022-03-08T22:51:38.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl restart docker || true 2022-03-08T22:51:39.761 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T22:51:39.768 INFO:tasks.kubeadm:172.21.15.42 is in 172.21.0.0/20 at pos 3881 2022-03-08T22:51:39.781 INFO:tasks.kubeadm:Pod subnet: 10.249.72.0/21 2022-03-08T22:51:39.781 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T22:51:39.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:51:39.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:51:39.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:51:39.810 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:39.810 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:39.810 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:39.811 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:39.811 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:39.818 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:39.819 INFO:teuthology.orchestra.run.smithi178.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:39.819 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:39.927 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:39.935 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:40.039 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:40.047 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:40.052 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:40.053 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:40.122 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:40.124 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:40.129 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:40.204 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:40.207 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:40.211 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:41.397 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:51:41.399 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:51:41.571 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:51:41.572 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:51:41.572 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:51:41.573 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:51:41.573 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:51:41.661 INFO:teuthology.orchestra.run.smithi178.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:41.664 INFO:teuthology.orchestra.run.smithi153.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:41.675 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:41.675 INFO:teuthology.orchestra.run.smithi178.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:41.675 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:41.677 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:41.677 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:41.678 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:41.728 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:51:41.729 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:51:41.750 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:51:41.750 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:51:41.840 INFO:teuthology.orchestra.run.smithi042.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:41.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:41.858 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:41.904 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:51:41.904 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:51:41.905 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:51:41.906 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:51:41.910 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:51:42.078 INFO:teuthology.orchestra.run.smithi178.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:51:42.079 INFO:teuthology.orchestra.run.smithi178.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:51:42.079 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2022-03-08T22:51:42.080 INFO:teuthology.orchestra.run.smithi178.stdout: apt-transport-https 2022-03-08T22:51:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:51:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:51:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2022-03-08T22:51:42.085 INFO:teuthology.orchestra.run.smithi153.stdout: apt-transport-https 2022-03-08T22:51:42.089 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:51:42.089 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:51:42.264 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:42.264 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 4,680 B of archives. 2022-03-08T22:51:42.265 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:51:42.265 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:51:42.273 INFO:teuthology.orchestra.run.smithi042.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:51:42.274 INFO:teuthology.orchestra.run.smithi042.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:51:42.274 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2022-03-08T22:51:42.275 INFO:teuthology.orchestra.run.smithi042.stdout: apt-transport-https 2022-03-08T22:51:42.282 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:42.282 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 4,680 B of archives. 2022-03-08T22:51:42.282 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:51:42.282 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:51:42.462 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:42.462 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 4,680 B of archives. 2022-03-08T22:51:42.462 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:51:42.463 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:51:42.497 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:42.498 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:42.498 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:42.513 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:42.513 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:42.513 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:42.519 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:42.519 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:42.520 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:42.525 INFO:teuthology.orchestra.run.smithi178.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:42.538 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:42.539 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:42.544 INFO:teuthology.orchestra.run.smithi153.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:42.575 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-08T22:51:42.594 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 4,680 B in 0s (24.8 kB/s) 2022-03-08T22:51:42.664 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:51:42.665 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:51:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:42.708 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:51:42.709 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:51:42.711 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:51:42.711 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:51:42.714 INFO:teuthology.orchestra.run.smithi042.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:42.755 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:51:42.763 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 4,680 B in 0s (26.2 kB/s) 2022-03-08T22:51:42.770 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:51:42.843 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:51:42.883 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:51:42.886 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:51:42.916 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:51:43.017 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:51:43.055 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:51:43.187 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:51:43.563 INFO:teuthology.orchestra.run.smithi178.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:51:43.577 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:43.577 INFO:teuthology.orchestra.run.smithi178.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:43.577 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:43.661 INFO:teuthology.orchestra.run.smithi153.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:51:43.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:43.675 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:43.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:43.754 INFO:teuthology.orchestra.run.smithi042.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:51:43.768 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:43.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:43.884 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:43.885 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:43.919 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:43.961 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:43.972 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:43.973 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:44.020 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:44.049 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:44.050 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:44.113 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:44.143 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:44.184 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:51:44.204 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:44.313 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:51:44.316 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:51:44.548 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:51:44.578 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:51:44.617 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:51:44.688 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 63.3 kB in 1s (65.4 kB/s) 2022-03-08T22:51:44.728 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 63.3 kB in 1s (78.3 kB/s) 2022-03-08T22:51:44.773 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 63.3 kB in 1s (67.2 kB/s) 2022-03-08T22:51:45.533 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:51:45.603 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:51:45.618 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:51:45.703 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:51:45.704 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:51:45.780 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:51:45.781 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:51:45.792 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:51:45.792 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:51:45.794 INFO:teuthology.orchestra.run.smithi178.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:45.810 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:45.810 INFO:teuthology.orchestra.run.smithi178.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:45.810 INFO:teuthology.orchestra.run.smithi178.stderr: 2022-03-08T22:51:45.865 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2022-03-08T22:51:45.874 INFO:teuthology.orchestra.run.smithi042.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:45.879 INFO:teuthology.orchestra.run.smithi153.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:45.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:45.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2022-03-08T22:51:45.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:45.892 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:45.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2022-03-08T22:51:45.944 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2022-03-08T22:51:45.945 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2022-03-08T22:51:46.040 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2022-03-08T22:51:46.040 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2022-03-08T22:51:46.108 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2022-03-08T22:51:46.109 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2022-03-08T22:51:46.109 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2022-03-08T22:51:46.110 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2022-03-08T22:51:46.189 INFO:teuthology.orchestra.run.smithi178.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:51:46.189 INFO:teuthology.orchestra.run.smithi178.stdout:bridge-utils set to manually installed. 2022-03-08T22:51:46.189 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2022-03-08T22:51:46.190 INFO:teuthology.orchestra.run.smithi178.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:51:46.190 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2022-03-08T22:51:46.191 INFO:teuthology.orchestra.run.smithi178.stdout: nftables 2022-03-08T22:51:46.238 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2022-03-08T22:51:46.239 INFO:teuthology.orchestra.run.smithi153.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:51:46.239 INFO:teuthology.orchestra.run.smithi153.stdout:bridge-utils set to manually installed. 2022-03-08T22:51:46.239 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2022-03-08T22:51:46.240 INFO:teuthology.orchestra.run.smithi178.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:51:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:51:46.241 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2022-03-08T22:51:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: nftables 2022-03-08T22:51:46.249 INFO:teuthology.orchestra.run.smithi042.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:51:46.249 INFO:teuthology.orchestra.run.smithi042.stdout:bridge-utils set to manually installed. 2022-03-08T22:51:46.250 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2022-03-08T22:51:46.250 INFO:teuthology.orchestra.run.smithi042.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:51:46.251 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2022-03-08T22:51:46.251 INFO:teuthology.orchestra.run.smithi042.stdout: nftables 2022-03-08T22:51:46.290 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2022-03-08T22:51:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:51:46.297 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2022-03-08T22:51:46.298 INFO:teuthology.orchestra.run.smithi042.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:51:46.445 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:46.445 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:51:46.445 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:51:46.445 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:51:46.492 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:46.492 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:51:46.492 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:51:46.492 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:51:46.495 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:46.495 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:51:46.496 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:51:46.496 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:51:46.540 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:51:46.581 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:51:46.617 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:51:46.664 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:51:46.685 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:51:46.687 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:51:46.756 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:51:46.824 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:51:46.826 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:51:47.034 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:51:47.433 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:51:47.950 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:51:48.078 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:51:48.506 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:51:48.519 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:51:48.697 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:51:48.733 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:51:48.873 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:51:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:49.114 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:49.121 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:49.122 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:49.122 INFO:teuthology.orchestra.run.smithi153.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:49.128 INFO:teuthology.orchestra.run.smithi153.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:49.152 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:49.153 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:49.153 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:49.160 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:49.160 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:49.160 INFO:teuthology.orchestra.run.smithi178.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:49.167 INFO:teuthology.orchestra.run.smithi178.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:49.175 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 73.6 MB in 3s (28.3 MB/s) 2022-03-08T22:51:49.215 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 73.6 MB in 3s (27.4 MB/s) 2022-03-08T22:51:49.271 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:51:49.312 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:51:49.314 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:51:49.318 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:51:49.356 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:51:49.359 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:51:49.361 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:51:49.411 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:51:49.689 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:51:49.703 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:51:49.713 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:51:49.726 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:51:49.739 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:51:49.763 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:51:50.001 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:51:50.446 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:51:50.624 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:51:50.825 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:51:50.832 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:51:50.837 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:51:50.845 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:51:50.891 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:51:50.899 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:51:51.089 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:51.089 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:51.090 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:51.103 INFO:teuthology.orchestra.run.smithi042.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:51.144 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:51:51.154 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 73.6 MB in 5s (16.1 MB/s) 2022-03-08T22:51:51.157 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:51:51.169 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:51:51.183 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:51:51.202 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:51.227 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:51.231 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:51:51.274 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:51:51.277 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:51:51.322 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:51:51.600 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:51:51.613 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:51:51.649 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:51:52.663 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2022-03-08T22:51:52.676 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:51:52.721 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:51:52.730 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package socat. 2022-03-08T22:51:52.742 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:51:52.761 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:51:52.774 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:51:52.779 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:51:52.828 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:51:53.089 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:51:53.102 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:51:53.121 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:51:53.133 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:51:53.138 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:53.170 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:51:53.185 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:51:53.193 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:51:53.251 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:51:54.624 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package socat. 2022-03-08T22:51:54.631 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:51:54.674 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:51:55.061 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:51:55.075 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:51:55.111 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:51:55.606 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:51:55.619 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:51:55.656 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:51:55.662 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:51:55.676 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:51:55.720 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:51:56.923 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:51:56.935 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:51:56.941 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:51:56.954 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:51:56.972 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:51:56.991 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:51:57.506 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:51:57.523 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:51:57.555 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:51:58.124 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:51:58.157 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:51:58.245 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:51:58.297 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:51:58.379 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:51:58.424 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:51:58.691 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:51:58.705 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:51:58.741 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:51:59.211 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:51:59.271 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:51:59.337 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:51:59.389 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:51:59.463 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:59.507 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:59.589 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:51:59.625 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:51:59.813 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:51:59.849 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:51:59.921 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:52:00.064 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:52:00.192 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:52:00.324 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:52:00.325 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:52:00.726 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:00.761 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:01.040 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:52:01.157 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:52:01.275 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:52:01.435 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:52:01.651 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:52:02.136 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:52:02.547 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:03.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:52:03.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:52:03.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:52:14.823 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:52:15.150 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:52:16.985 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:52:17.958 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:52:18.676 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:52:20.189 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:52:20.405 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:52:21.174 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:52:23.091 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:52:23.671 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:52:25.291 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:52:25.329 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:52:26.849 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:52:27.034 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:52:28.575 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:52:30.703 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:52:32.605 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:52:32.861 INFO:teuthology.orchestra.run.smithi178.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:52:34.539 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:52:34.864 INFO:teuthology.orchestra.run.smithi153.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:52:36.858 INFO:teuthology.orchestra.run.smithi042.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:52:36.862 INFO:tasks.kubeadm:Token: abcdef.n9k5781sd9oo3gyo 2022-03-08T22:52:36.863 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): '172.21.15.42', Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): '172.21.15.153', Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): '172.21.15.178'} 2022-03-08T22:52:36.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo kubeadm init --node-name smithi042 --token abcdef.n9k5781sd9oo3gyo --pod-network-cidr 10.249.72.0/21 2022-03-08T22:52:37.059 INFO:teuthology.orchestra.run.smithi042.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T22:52:37.059 INFO:teuthology.orchestra.run.smithi042.stdout:[preflight] Running pre-flight checks 2022-03-08T22:52:37.752 INFO:teuthology.orchestra.run.smithi042.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T22:52:37.752 INFO:teuthology.orchestra.run.smithi042.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T22:52:37.752 INFO:teuthology.orchestra.run.smithi042.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T22:52:38.079 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T22:52:38.596 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "ca" certificate and key 2022-03-08T22:52:38.942 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T22:52:38.942 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi042] and IPs [10.96.0.1 172.21.15.42] 2022-03-08T22:52:39.107 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T22:52:39.299 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T22:52:39.402 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T22:52:39.600 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T22:52:39.661 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T22:52:39.661 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi042] and IPs [172.21.15.42 127.0.0.1 ::1] 2022-03-08T22:52:39.758 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T22:52:39.758 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi042] and IPs [172.21.15.42 127.0.0.1 ::1] 2022-03-08T22:52:39.839 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T22:52:39.897 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T22:52:40.233 INFO:teuthology.orchestra.run.smithi042.stdout:[certs] Generating "sa" key and public key 2022-03-08T22:52:40.233 INFO:teuthology.orchestra.run.smithi042.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T22:52:40.339 INFO:teuthology.orchestra.run.smithi042.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T22:52:40.641 INFO:teuthology.orchestra.run.smithi042.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T22:52:40.944 INFO:teuthology.orchestra.run.smithi042.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T22:52:41.134 INFO:teuthology.orchestra.run.smithi042.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T22:52:41.148 INFO:teuthology.orchestra.run.smithi042.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:52:41.149 INFO:teuthology.orchestra.run.smithi042.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:52:41.150 INFO:teuthology.orchestra.run.smithi042.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:52:41.351 INFO:teuthology.orchestra.run.smithi042.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T22:52:41.351 INFO:teuthology.orchestra.run.smithi042.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T22:52:41.355 INFO:teuthology.orchestra.run.smithi042.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T22:52:41.356 INFO:teuthology.orchestra.run.smithi042.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T22:52:41.357 INFO:teuthology.orchestra.run.smithi042.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T22:52:41.359 INFO:teuthology.orchestra.run.smithi042.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T22:52:58.367 INFO:teuthology.orchestra.run.smithi042.stdout:[apiclient] All control plane components are healthy after 17.007866 seconds 2022-03-08T22:52:58.367 INFO:teuthology.orchestra.run.smithi042.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T22:52:58.384 INFO:teuthology.orchestra.run.smithi042.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T22:52:58.384 INFO:teuthology.orchestra.run.smithi042.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T22:52:59.010 INFO:teuthology.orchestra.run.smithi042.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T22:52:59.010 INFO:teuthology.orchestra.run.smithi042.stdout:[mark-control-plane] Marking the node smithi042 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T22:52:59.011 INFO:teuthology.orchestra.run.smithi042.stdout:[mark-control-plane] Marking the node smithi042 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T22:52:59.536 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] Using token: abcdef.n9k5781sd9oo3gyo 2022-03-08T22:52:59.537 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T22:52:59.566 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T22:52:59.604 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T22:52:59.672 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T22:52:59.674 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T22:52:59.684 INFO:teuthology.orchestra.run.smithi042.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T22:52:59.726 INFO:teuthology.orchestra.run.smithi042.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T22:53:00.218 INFO:teuthology.orchestra.run.smithi042.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T22:53:00.351 INFO:teuthology.orchestra.run.smithi042.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T22:53:00.352 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.353 INFO:teuthology.orchestra.run.smithi042.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T22:53:00.353 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.353 INFO:teuthology.orchestra.run.smithi042.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T22:53:00.353 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.354 INFO:teuthology.orchestra.run.smithi042.stdout: mkdir -p $HOME/.kube 2022-03-08T22:53:00.354 INFO:teuthology.orchestra.run.smithi042.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T22:53:00.354 INFO:teuthology.orchestra.run.smithi042.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T22:53:00.354 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.354 INFO:teuthology.orchestra.run.smithi042.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T22:53:00.355 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.355 INFO:teuthology.orchestra.run.smithi042.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T22:53:00.355 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.355 INFO:teuthology.orchestra.run.smithi042.stdout:You should now deploy a pod network to the cluster. 2022-03-08T22:53:00.355 INFO:teuthology.orchestra.run.smithi042.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T22:53:00.356 INFO:teuthology.orchestra.run.smithi042.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T22:53:00.356 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.356 INFO:teuthology.orchestra.run.smithi042.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T22:53:00.356 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:53:00.356 INFO:teuthology.orchestra.run.smithi042.stdout:kubeadm join 172.21.15.42:6443 --token abcdef.n9k5781sd9oo3gyo \ 2022-03-08T22:53:00.357 INFO:teuthology.orchestra.run.smithi042.stdout: --discovery-token-ca-cert-hash sha256:6627b905d0326fcb62a0ae9e25f9ccedacecbb970da5f37f33671264beccb84c 2022-03-08T22:53:00.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo kubeadm join 172.21.15.42:6443 --node-name smithi153 --token abcdef.n9k5781sd9oo3gyo --discovery-token-unsafe-skip-ca-verification 2022-03-08T22:53:00.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo kubeadm join 172.21.15.42:6443 --node-name smithi178 --token abcdef.n9k5781sd9oo3gyo --discovery-token-unsafe-skip-ca-verification 2022-03-08T22:53:00.397 INFO:teuthology.orchestra.run.smithi153.stdout:[preflight] Running pre-flight checks 2022-03-08T22:53:00.404 INFO:teuthology.orchestra.run.smithi178.stdout:[preflight] Running pre-flight checks 2022-03-08T22:53:13.151 INFO:teuthology.orchestra.run.smithi153.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T22:53:13.152 INFO:teuthology.orchestra.run.smithi153.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T22:53:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:W0308 22:53:13.511747 10626 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T22:53:13.528 INFO:teuthology.orchestra.run.smithi153.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:53:13.528 INFO:teuthology.orchestra.run.smithi153.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:53:13.529 INFO:teuthology.orchestra.run.smithi153.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:53:13.750 INFO:teuthology.orchestra.run.smithi153.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T22:53:14.011 INFO:teuthology.orchestra.run.smithi178.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T22:53:14.012 INFO:teuthology.orchestra.run.smithi178.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T22:53:14.388 INFO:teuthology.orchestra.run.smithi178.stderr:W0308 22:53:14.390477 10685 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T22:53:14.436 INFO:teuthology.orchestra.run.smithi178.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:53:14.436 INFO:teuthology.orchestra.run.smithi178.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:53:14.436 INFO:teuthology.orchestra.run.smithi178.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:53:14.671 INFO:teuthology.orchestra.run.smithi178.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T22:53:20.212 INFO:teuthology.orchestra.run.smithi178.stdout: 2022-03-08T22:53:20.212 INFO:teuthology.orchestra.run.smithi178.stdout:This node has joined the cluster: 2022-03-08T22:53:20.213 INFO:teuthology.orchestra.run.smithi178.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T22:53:20.213 INFO:teuthology.orchestra.run.smithi178.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T22:53:20.214 INFO:teuthology.orchestra.run.smithi178.stdout: 2022-03-08T22:53:20.214 INFO:teuthology.orchestra.run.smithi178.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T22:53:20.215 INFO:teuthology.orchestra.run.smithi178.stdout: 2022-03-08T22:53:27.273 INFO:teuthology.orchestra.run.smithi153.stdout: 2022-03-08T22:53:27.273 INFO:teuthology.orchestra.run.smithi153.stdout:This node has joined the cluster: 2022-03-08T22:53:27.273 INFO:teuthology.orchestra.run.smithi153.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T22:53:27.274 INFO:teuthology.orchestra.run.smithi153.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T22:53:27.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2022-03-08T22:53:27.274 INFO:teuthology.orchestra.run.smithi153.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T22:53:27.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2022-03-08T22:53:27.280 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:27.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T22:53:27.296 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T22:53:27.296 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:53:27.358 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:53:27.375 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:53:27.393 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:27.394 DEBUG:teuthology.orchestra.run.smithi042:> dd of=.kube/config 2022-03-08T22:53:27.401 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:27.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/root/.kube/config 2022-03-08T22:53:27.416 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:27.417 DEBUG:teuthology.orchestra.run.smithi153:> dd of=.kube/config 2022-03-08T22:53:27.424 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:27.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/root/.kube/config 2022-03-08T22:53:27.439 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:27.440 DEBUG:teuthology.orchestra.run.smithi178:> dd of=.kube/config 2022-03-08T22:53:27.447 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:27.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/root/.kube/config 2022-03-08T22:53:27.503 DEBUG:teuthology.orchestra.run.smithi042:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-08T22:53:27.517 INFO:teuthology.orchestra.run.smithi042.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-08T22:53:27.518 INFO:teuthology.orchestra.run.smithi042.stderr: Dload Upload Total Spent Left Speed 2022-03-08T22:53:27.638 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.639 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: policy/v1beta1 2022-03-08T22:53:27.639 INFO:teuthology.orchestra.run.smithi042.stdout:kind: PodSecurityPolicy 2022-03-08T22:53:27.639 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.639 INFO:teuthology.orchestra.run.smithi042.stdout: name: psp.flannel.unprivileged 2022-03-08T22:53:27.640 INFO:teuthology.orchestra.run.smithi042.stdout: annotations: 2022-03-08T22:53:27.640 INFO:teuthology.orchestra.run.smithi042.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-08T22:53:27.640 INFO:teuthology.orchestra.run.smithi042.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-08T22:53:27.640 INFO:teuthology.orchestra.run.smithi042.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-08T22:53:27.641 INFO:teuthology.orchestra.run.smithi042.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-08T22:53:27.641 INFO:teuthology.orchestra.run.smithi042.stdout:spec: 2022-03-08T22:53:27.641 INFO:teuthology.orchestra.run.smithi042.stdout: privileged: false 2022-03-08T22:53:27.641 INFO:teuthology.orchestra.run.smithi042.stdout: volumes: 2022-03-08T22:53:27.642 INFO:teuthology.orchestra.run.smithi042.stdout: - configMap 2022-03-08T22:53:27.642 INFO:teuthology.orchestra.run.smithi042.stdout: - secret 2022-03-08T22:53:27.642 INFO:teuthology.orchestra.run.smithi042.stdout: - emptyDir 2022-03-08T22:53:27.642 INFO:teuthology.orchestra.run.smithi042.stdout: - hostPath 2022-03-08T22:53:27.642 INFO:teuthology.orchestra.run.smithi042.stdout: allowedHostPaths: 2022-03-08T22:53:27.643 INFO:teuthology.orchestra.run.smithi042.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-08T22:53:27.643 INFO:teuthology.orchestra.run.smithi042.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-08T22:53:27.643 INFO:teuthology.orchestra.run.smithi042.stdout: - pathPrefix: "/run/flannel" 2022-03-08T22:53:27.643 INFO:teuthology.orchestra.run.smithi042.stdout: readOnlyRootFilesystem: false 2022-03-08T22:53:27.644 INFO:teuthology.orchestra.run.smithi042.stdout: # Users and groups 2022-03-08T22:53:27.644 INFO:teuthology.orchestra.run.smithi042.stdout: runAsUser: 2022-03-08T22:53:27.644 INFO:teuthology.orchestra.run.smithi042.stdout: rule: RunAsAny 2022-03-08T22:53:27.644 INFO:teuthology.orchestra.run.smithi042.stdout: supplementalGroups: 2022-03-08T22:53:27.644 INFO:teuthology.orchestra.run.smithi042.stdout: rule: RunAsAny 2022-03-08T22:53:27.645 INFO:teuthology.orchestra.run.smithi042.stdout: fsGroup: 2022-03-08T22:53:27.645 INFO:teuthology.orchestra.run.smithi042.stdout: rule: RunAsAny 2022-03-08T22:53:27.645 INFO:teuthology.orchestra.run.smithi042.stdout: # Privilege Escalation 2022-03-08T22:53:27.646 INFO:teuthology.orchestra.run.smithi042.stdout: allowPrivilegeEscalation: false 2022-03-08T22:53:27.646 INFO:teuthology.orchestra.run.smithi042.stdout: defaultAllowPrivilegeEscalation: false 2022-03-08T22:53:27.647 INFO:teuthology.orchestra.run.smithi042.stdout: # Capabilities 2022-03-08T22:53:27.647 INFO:teuthology.orchestra.run.smithi042.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-08T22:53:27.647 INFO:teuthology.orchestra.run.smithi042.stdout: defaultAddCapabilities: [] 2022-03-08T22:53:27.648 INFO:teuthology.orchestra.run.smithi042.stdout: requiredDropCapabilities: [] 2022-03-08T22:53:27.648 INFO:teuthology.orchestra.run.smithi042.stdout: # Host namespaces 2022-03-08T22:53:27.648 INFO:teuthology.orchestra.run.smithi042.stdout: hostPID: false 2022-03-08T22:53:27.648 INFO:teuthology.orchestra.run.smithi042.stdout: hostIPC: false 2022-03-08T22:53:27.648 INFO:teuthology.orchestra.run.smithi042.stdout: hostNetwork: true 2022-03-08T22:53:27.649 INFO:teuthology.orchestra.run.smithi042.stdout: hostPorts: 2022-03-08T22:53:27.649 INFO:teuthology.orchestra.run.smithi042.stdout: - min: 0 2022-03-08T22:53:27.649 INFO:teuthology.orchestra.run.smithi042.stdout: max: 65535 2022-03-08T22:53:27.649 INFO:teuthology.orchestra.run.smithi042.stdout: # SELinux 2022-03-08T22:53:27.650 INFO:teuthology.orchestra.run.smithi042.stdout: seLinux: 2022-03-08T22:53:27.650 INFO:teuthology.orchestra.run.smithi042.stdout: # SELinux is unused in CaaSP 2022-03-08T22:53:27.650 INFO:teuthology.orchestra.run.smithi042.stdout: rule: 'RunAsAny' 2022-03-08T22:53:27.650 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.651 INFO:teuthology.orchestra.run.smithi042.stdout:kind: ClusterRole 2022-03-08T22:53:27.651 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-08T22:53:27.651 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.651 INFO:teuthology.orchestra.run.smithi042.stdout: name: flannel 2022-03-08T22:53:27.652 INFO:teuthology.orchestra.run.smithi042.stdout:rules: 2022-03-08T22:53:27.652 INFO:teuthology.orchestra.run.smithi042.stdout:- apiGroups: ['extensions'] 2022-03-08T22:53:27.652 INFO:teuthology.orchestra.run.smithi042.stdout: resources: ['podsecuritypolicies'] 2022-03-08T22:53:27.652 INFO:teuthology.orchestra.run.smithi042.stdout: verbs: ['use'] 2022-03-08T22:53:27.652 INFO:teuthology.orchestra.run.smithi042.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-08T22:53:27.653 INFO:teuthology.orchestra.run.smithi042.stdout:- apiGroups: 2022-03-08T22:53:27.653 INFO:teuthology.orchestra.run.smithi042.stdout: - "" 2022-03-08T22:53:27.653 INFO:teuthology.orchestra.run.smithi042.stdout: resources: 2022-03-08T22:53:27.653 INFO:teuthology.orchestra.run.smithi042.stdout: - pods 2022-03-08T22:53:27.654 INFO:teuthology.orchestra.run.smithi042.stdout: verbs: 2022-03-08T22:53:27.654 INFO:teuthology.orchestra.run.smithi042.stdout: - get 2022-03-08T22:53:27.654 INFO:teuthology.orchestra.run.smithi042.stdout:- apiGroups: 2022-03-08T22:53:27.654 INFO:teuthology.orchestra.run.smithi042.stdout: - "" 2022-03-08T22:53:27.655 INFO:teuthology.orchestra.run.smithi042.stdout: resources: 2022-03-08T22:53:27.655 INFO:teuthology.orchestra.run.smithi042.stdout: - nodes 2022-03-08T22:53:27.655 INFO:teuthology.orchestra.run.smithi042.stdout: verbs: 2022-03-08T22:53:27.655 INFO:teuthology.orchestra.run.smithi042.stdout: - list 2022-03-08T22:53:27.655 INFO:teuthology.orchestra.run.smithi042.stdout: - watch 2022-03-08T22:53:27.656 INFO:teuthology.orchestra.run.smithi042.stdout:- apiGroups: 2022-03-08T22:53:27.656 INFO:teuthology.orchestra.run.smithi042.stdout: - "" 2022-03-08T22:53:27.656 INFO:teuthology.orchestra.run.smithi042.stdout: resources: 2022-03-08T22:53:27.656 INFO:teuthology.orchestra.run.smithi042.stdout: - nodes/status 2022-03-08T22:53:27.657 INFO:teuthology.orchestra.run.smithi042.stdout: verbs: 2022-03-08T22:53:27.657 INFO:teuthology.orchestra.run.smithi042.stdout: - patch 2022-03-08T22:53:27.657 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.657 INFO:teuthology.orchestra.run.smithi042.stdout:kind: ClusterRoleBinding 2022-03-08T22:53:27.657 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-08T22:53:27.658 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.658 INFO:teuthology.orchestra.run.smithi042.stdout: name: flannel 2022-03-08T22:53:27.658 INFO:teuthology.orchestra.run.smithi042.stdout:roleRef: 2022-03-08T22:53:27.658 INFO:teuthology.orchestra.run.smithi042.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-08T22:53:27.659 INFO:teuthology.orchestra.run.smithi042.stdout: kind: ClusterRole 2022-03-08T22:53:27.659 INFO:teuthology.orchestra.run.smithi042.stdout: name: flannel 2022-03-08T22:53:27.659 INFO:teuthology.orchestra.run.smithi042.stdout:subjects: 2022-03-08T22:53:27.659 INFO:teuthology.orchestra.run.smithi042.stdout:- kind: ServiceAccount 2022-03-08T22:53:27.659 INFO:teuthology.orchestra.run.smithi042.stdout: name: flannel 2022-03-08T22:53:27.660 INFO:teuthology.orchestra.run.smithi042.stdout: namespace: kube-system 2022-03-08T22:53:27.660 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.660 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: v1 2022-03-08T22:53:27.660 INFO:teuthology.orchestra.run.smithi042.stdout:kind: ServiceAccount 2022-03-08T22:53:27.661 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.661 INFO:teuthology.orchestra.run.smithi042.stdout: name: flannel 2022-03-08T22:53:27.661 INFO:teuthology.orchestra.run.smithi042.stdout: namespace: kube-system 2022-03-08T22:53:27.661 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.661 INFO:teuthology.orchestra.run.smithi042.stdout:kind: ConfigMap 2022-03-08T22:53:27.662 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: v1 2022-03-08T22:53:27.662 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.662 INFO:teuthology.orchestra.run.smithi042.stdout: name: kube-flannel-cfg 2022-03-08T22:53:27.662 INFO:teuthology.orchestra.run.smithi042.stdout: namespace: kube-system 2022-03-08T22:53:27.663 INFO:teuthology.orchestra.run.smithi042.stdout: labels: 2022-03-08T22:53:27.663 INFO:teuthology.orchestra.run.smithi042.stdout: tier: node 2022-03-08T22:53:27.663 INFO:teuthology.orchestra.run.smithi042.stdout: app: flannel 2022-03-08T22:53:27.663 INFO:teuthology.orchestra.run.smithi042.stdout:data: 2022-03-08T22:53:27.664 INFO:teuthology.orchestra.run.smithi042.stdout: cni-conf.json: | 2022-03-08T22:53:27.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2022-03-08T22:53:27.664 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "cbr0", 2022-03-08T22:53:27.664 INFO:teuthology.orchestra.run.smithi042.stdout: "cniVersion": "0.3.1", 2022-03-08T22:53:27.664 INFO:teuthology.orchestra.run.smithi042.stdout: "plugins": [ 2022-03-08T22:53:27.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2022-03-08T22:53:27.665 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "flannel", 2022-03-08T22:53:27.665 INFO:teuthology.orchestra.run.smithi042.stdout: "delegate": { 2022-03-08T22:53:27.665 INFO:teuthology.orchestra.run.smithi042.stdout: "hairpinMode": true, 2022-03-08T22:53:27.666 INFO:teuthology.orchestra.run.smithi042.stdout: "isDefaultGateway": true 2022-03-08T22:53:27.666 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2022-03-08T22:53:27.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2022-03-08T22:53:27.666 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "portmap", 2022-03-08T22:53:27.667 INFO:teuthology.orchestra.run.smithi042.stdout: "capabilities": { 2022-03-08T22:53:27.667 INFO:teuthology.orchestra.run.smithi042.stdout: "portMappings": true 2022-03-08T22:53:27.667 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.667 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.668 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2022-03-08T22:53:27.668 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.668 INFO:teuthology.orchestra.run.smithi042.stdout: net-conf.json: | 2022-03-08T22:53:27.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2022-03-08T22:53:27.668 INFO:teuthology.orchestra.run.smithi042.stdout: "Network": "10.244.0.0/16", 2022-03-08T22:53:27.669 INFO:teuthology.orchestra.run.smithi042.stdout: "Backend": { 2022-03-08T22:53:27.669 INFO:teuthology.orchestra.run.smithi042.stdout: "Type": "vxlan" 2022-03-08T22:53:27.669 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.669 INFO:teuthology.orchestra.run.smithi042.stdout: } 2022-03-08T22:53:27.670 INFO:teuthology.orchestra.run.smithi042.stdout:--- 2022-03-08T22:53:27.670 INFO:teuthology.orchestra.run.smithi042.stdout:apiVersion: apps/v1 2022-03-08T22:53:27.670 INFO:teuthology.orchestra.run.smithi042.stdout:kind: DaemonSet 2022-03-08T22:53:27.670 INFO:teuthology.orchestra.run.smithi042.stdout:metadata: 2022-03-08T22:53:27.670 INFO:teuthology.orchestra.run.smithi042.stdout: name: kube-flannel-ds 2022-03-08T22:53:27.671 INFO:teuthology.orchestra.run.smithi042.stdout: namespace: kube-system 2022-03-08T22:53:27.671 INFO:teuthology.orchestra.run.smithi042.stdout: labels: 2022-03-08T22:53:27.671 INFO:teuthology.orchestra.run.smithi042.stdout: tier: node 2022-03-08T22:53:27.671 INFO:teuthology.orchestra.run.smithi042.stdout: app: flannel 2022-03-08T22:53:27.672 INFO:teuthology.orchestra.run.smithi042.stdout:spec: 2022-03-08T22:53:27.672 INFO:teuthology.orchestra.run.smithi042.stdout: selector: 2022-03-08T22:53:27.672 INFO:teuthology.orchestra.run.smithi042.stdout: matchLabels: 2022-03-08T22:53:27.672 INFO:teuthology.orchestra.run.smithi042.stdout: app: flannel 2022-03-08T22:53:27.672 INFO:teuthology.orchestra.run.smithi042.stdout: template: 2022-03-08T22:53:27.673 INFO:teuthology.orchestra.run.smithi042.stdout: metadata: 2022-03-08T22:53:27.673 INFO:teuthology.orchestra.run.smithi042.stdout: labels: 2022-03-08T22:53:27.673 INFO:teuthology.orchestra.run.smithi042.stdout: tier: node 2022-03-08T22:53:27.673 INFO:teuthology.orchestra.run.smithi042.stdout: app: flannel 2022-03-08T22:53:27.674 INFO:teuthology.orchestra.run.smithi042.stdout: spec: 2022-03-08T22:53:27.674 INFO:teuthology.orchestra.run.smithi042.stdout: affinity: 2022-03-08T22:53:27.674 INFO:teuthology.orchestra.run.smithi042.stdout: nodeAffinity: 2022-03-08T22:53:27.674 INFO:teuthology.orchestra.run.smithi042.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-08T22:53:27.674 INFO:teuthology.orchestra.run.smithi042.stdout: nodeSelectorTerms: 2022-03-08T22:53:27.675 INFO:teuthology.orchestra.run.smithi042.stdout: - matchExpressions: 2022-03-08T22:53:27.675 INFO:teuthology.orchestra.run.smithi042.stdout: - key: kubernetes.io/os 2022-03-08T22:53:27.675 INFO:teuthology.orchestra.run.smithi042.stdout: operator: In 2022-03-08T22:53:27.675 INFO:teuthology.orchestra.run.smithi042.stdout: values: 2022-03-08T22:53:27.676 INFO:teuthology.orchestra.run.smithi042.stdout: - linux 2022-03-08T22:53:27.676 INFO:teuthology.orchestra.run.smithi042.stdout: hostNetwork: true 2022-03-08T22:53:27.676 INFO:teuthology.orchestra.run.smithi042.stdout: priorityClassName: system-node-critical 2022-03-08T22:53:27.676 INFO:teuthology.orchestra.run.smithi042.stdout: tolerations: 2022-03-08T22:53:27.677 INFO:teuthology.orchestra.run.smithi042.stdout: - operator: Exists 2022-03-08T22:53:27.677 INFO:teuthology.orchestra.run.smithi042.stdout: effect: NoSchedule 2022-03-08T22:53:27.677 INFO:teuthology.orchestra.run.smithi042.stdout: serviceAccountName: flannel 2022-03-08T22:53:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: initContainers: 2022-03-08T22:53:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: - name: install-cni-plugin 2022-03-08T22:53:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T22:53:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-08T22:53:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: command: 2022-03-08T22:53:27.679 INFO:teuthology.orchestra.run.smithi042.stdout: - cp 2022-03-08T22:53:27.679 INFO:teuthology.orchestra.run.smithi042.stdout: args: 2022-03-08T22:53:27.679 INFO:teuthology.orchestra.run.smithi042.stdout: - -f 2022-03-08T22:53:27.679 INFO:teuthology.orchestra.run.smithi042.stdout: - /flannel 2022-03-08T22:53:27.680 INFO:teuthology.orchestra.run.smithi042.stdout: - /opt/cni/bin/flannel 2022-03-08T22:53:27.680 INFO:teuthology.orchestra.run.smithi042.stdout: volumeMounts: 2022-03-08T22:53:27.680 INFO:teuthology.orchestra.run.smithi042.stdout: - name: cni-plugin 2022-03-08T22:53:27.680 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /opt/cni/bin 2022-03-08T22:53:27.680 INFO:teuthology.orchestra.run.smithi042.stdout: - name: install-cni 2022-03-08T22:53:27.681 INFO:teuthology.orchestra.run.smithi042.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T22:53:27.681 INFO:teuthology.orchestra.run.smithi042.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-08T22:53:27.681 INFO:teuthology.orchestra.run.smithi042.stdout: command: 2022-03-08T22:53:27.681 INFO:teuthology.orchestra.run.smithi042.stdout: - cp 2022-03-08T22:53:27.682 INFO:teuthology.orchestra.run.smithi042.stdout: args: 2022-03-08T22:53:27.682 INFO:teuthology.orchestra.run.smithi042.stdout: - -f 2022-03-08T22:53:27.682 INFO:teuthology.orchestra.run.smithi042.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-08T22:53:27.683 INFO:teuthology.orchestra.run.smithi042.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 47041 0 --:--:-- --:--:-- --:--:-- 47041 2022-03-08T22:53:27.684 INFO:teuthology.orchestra.run.smithi042.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-08T22:53:27.684 INFO:teuthology.orchestra.run.smithi042.stdout: volumeMounts: 2022-03-08T22:53:27.684 INFO:teuthology.orchestra.run.smithi042.stdout: - name: cni 2022-03-08T22:53:27.685 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /etc/cni/net.d 2022-03-08T22:53:27.685 INFO:teuthology.orchestra.run.smithi042.stdout: - name: flannel-cfg 2022-03-08T22:53:27.685 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /etc/kube-flannel/ 2022-03-08T22:53:27.685 INFO:teuthology.orchestra.run.smithi042.stdout: containers: 2022-03-08T22:53:27.685 INFO:teuthology.orchestra.run.smithi042.stdout: - name: kube-flannel 2022-03-08T22:53:27.686 INFO:teuthology.orchestra.run.smithi042.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T22:53:27.686 INFO:teuthology.orchestra.run.smithi042.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-08T22:53:27.686 INFO:teuthology.orchestra.run.smithi042.stdout: command: 2022-03-08T22:53:27.686 INFO:teuthology.orchestra.run.smithi042.stdout: - /opt/bin/flanneld 2022-03-08T22:53:27.687 INFO:teuthology.orchestra.run.smithi042.stdout: args: 2022-03-08T22:53:27.687 INFO:teuthology.orchestra.run.smithi042.stdout: - --ip-masq 2022-03-08T22:53:27.687 INFO:teuthology.orchestra.run.smithi042.stdout: - --kube-subnet-mgr 2022-03-08T22:53:27.687 INFO:teuthology.orchestra.run.smithi042.stdout: resources: 2022-03-08T22:53:27.688 INFO:teuthology.orchestra.run.smithi042.stdout: requests: 2022-03-08T22:53:27.688 INFO:teuthology.orchestra.run.smithi042.stdout: cpu: "100m" 2022-03-08T22:53:27.688 INFO:teuthology.orchestra.run.smithi042.stdout: memory: "50Mi" 2022-03-08T22:53:27.689 INFO:teuthology.orchestra.run.smithi042.stdout: limits: 2022-03-08T22:53:27.689 INFO:teuthology.orchestra.run.smithi042.stdout: cpu: "100m" 2022-03-08T22:53:27.690 INFO:teuthology.orchestra.run.smithi042.stdout: memory: "50Mi" 2022-03-08T22:53:27.690 INFO:teuthology.orchestra.run.smithi042.stdout: securityContext: 2022-03-08T22:53:27.691 INFO:teuthology.orchestra.run.smithi042.stdout: privileged: false 2022-03-08T22:53:27.691 INFO:teuthology.orchestra.run.smithi042.stdout: capabilities: 2022-03-08T22:53:27.691 INFO:teuthology.orchestra.run.smithi042.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-08T22:53:27.691 INFO:teuthology.orchestra.run.smithi042.stdout: env: 2022-03-08T22:53:27.691 INFO:teuthology.orchestra.run.smithi042.stdout: - name: POD_NAME 2022-03-08T22:53:27.692 INFO:teuthology.orchestra.run.smithi042.stdout: valueFrom: 2022-03-08T22:53:27.692 INFO:teuthology.orchestra.run.smithi042.stdout: fieldRef: 2022-03-08T22:53:27.692 INFO:teuthology.orchestra.run.smithi042.stdout: fieldPath: metadata.name 2022-03-08T22:53:27.692 INFO:teuthology.orchestra.run.smithi042.stdout: - name: POD_NAMESPACE 2022-03-08T22:53:27.693 INFO:teuthology.orchestra.run.smithi042.stdout: valueFrom: 2022-03-08T22:53:27.693 INFO:teuthology.orchestra.run.smithi042.stdout: fieldRef: 2022-03-08T22:53:27.693 INFO:teuthology.orchestra.run.smithi042.stdout: fieldPath: metadata.namespace 2022-03-08T22:53:27.693 INFO:teuthology.orchestra.run.smithi042.stdout: volumeMounts: 2022-03-08T22:53:27.694 INFO:teuthology.orchestra.run.smithi042.stdout: - name: run 2022-03-08T22:53:27.694 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /run/flannel 2022-03-08T22:53:27.694 INFO:teuthology.orchestra.run.smithi042.stdout: - name: flannel-cfg 2022-03-08T22:53:27.694 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /etc/kube-flannel/ 2022-03-08T22:53:27.694 INFO:teuthology.orchestra.run.smithi042.stdout: - name: xtables-lock 2022-03-08T22:53:27.695 INFO:teuthology.orchestra.run.smithi042.stdout: mountPath: /run/xtables.lock 2022-03-08T22:53:27.695 INFO:teuthology.orchestra.run.smithi042.stdout: volumes: 2022-03-08T22:53:27.695 INFO:teuthology.orchestra.run.smithi042.stdout: - name: run 2022-03-08T22:53:27.695 INFO:teuthology.orchestra.run.smithi042.stdout: hostPath: 2022-03-08T22:53:27.696 INFO:teuthology.orchestra.run.smithi042.stdout: path: /run/flannel 2022-03-08T22:53:27.696 INFO:teuthology.orchestra.run.smithi042.stdout: - name: cni-plugin 2022-03-08T22:53:27.696 INFO:teuthology.orchestra.run.smithi042.stdout: hostPath: 2022-03-08T22:53:27.696 INFO:teuthology.orchestra.run.smithi042.stdout: path: /opt/cni/bin 2022-03-08T22:53:27.696 INFO:teuthology.orchestra.run.smithi042.stdout: - name: cni 2022-03-08T22:53:27.697 INFO:teuthology.orchestra.run.smithi042.stdout: hostPath: 2022-03-08T22:53:27.697 INFO:teuthology.orchestra.run.smithi042.stdout: path: /etc/cni/net.d 2022-03-08T22:53:27.697 INFO:teuthology.orchestra.run.smithi042.stdout: - name: flannel-cfg 2022-03-08T22:53:27.697 INFO:teuthology.orchestra.run.smithi042.stdout: configMap: 2022-03-08T22:53:27.698 INFO:teuthology.orchestra.run.smithi042.stdout: name: kube-flannel-cfg 2022-03-08T22:53:27.698 INFO:teuthology.orchestra.run.smithi042.stdout: - name: xtables-lock 2022-03-08T22:53:27.698 INFO:teuthology.orchestra.run.smithi042.stdout: hostPath: 2022-03-08T22:53:27.698 INFO:teuthology.orchestra.run.smithi042.stdout: path: /run/xtables.lock 2022-03-08T22:53:27.698 INFO:teuthology.orchestra.run.smithi042.stdout: type: FileOrCreate 2022-03-08T22:53:27.726 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-08T22:53:27.726 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.249.72.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-08T22:53:27.740 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.249.72.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-08T22:53:27.741 DEBUG:teuthology.orchestra.run.smithi042:> kubectl apply -f - 2022-03-08T22:53:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:53:30.124 INFO:teuthology.orchestra.run.smithi042.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-08T22:53:30.152 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-08T22:53:30.161 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-08T22:53:30.171 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/flannel created 2022-03-08T22:53:30.242 INFO:teuthology.orchestra.run.smithi042.stdout:configmap/kube-flannel-cfg created 2022-03-08T22:53:30.281 INFO:teuthology.orchestra.run.smithi042.stdout:daemonset.apps/kube-flannel-ds created 2022-03-08T22:53:30.286 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T22:53:30.287 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:30.287 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:30.295 INFO:tasks.kubeadm:Scratch devices on smithi042: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:30.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:53:30.356 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records in 2022-03-08T22:53:30.357 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2022-03-08T22:53:30.357 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010746 s, 976 MB/s 2022-03-08T22:53:30.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:53:30.380 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records in 2022-03-08T22:53:30.380 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2022-03-08T22:53:30.381 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00983679 s, 1.1 GB/s 2022-03-08T22:53:30.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:53:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records in 2022-03-08T22:53:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2022-03-08T22:53:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00931847 s, 1.1 GB/s 2022-03-08T22:53:30.405 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:53:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records in 2022-03-08T22:53:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2022-03-08T22:53:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00975691 s, 1.1 GB/s 2022-03-08T22:53:30.431 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:30.432 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:30.440 INFO:tasks.kubeadm:Scratch devices on smithi153: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:30.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:53:30.467 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records in 2022-03-08T22:53:30.467 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2022-03-08T22:53:30.467 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155328 s, 675 MB/s 2022-03-08T22:53:30.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:53:30.494 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records in 2022-03-08T22:53:30.495 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2022-03-08T22:53:30.495 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164751 s, 636 MB/s 2022-03-08T22:53:30.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:53:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records in 2022-03-08T22:53:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2022-03-08T22:53:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.020614 s, 509 MB/s 2022-03-08T22:53:30.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:53:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records in 2022-03-08T22:53:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2022-03-08T22:53:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156347 s, 671 MB/s 2022-03-08T22:53:30.609 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:30.609 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:30.615 INFO:tasks.kubeadm:Scratch devices on smithi178: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:30.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:53:30.643 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records in 2022-03-08T22:53:30.644 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2022-03-08T22:53:30.644 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156502 s, 670 MB/s 2022-03-08T22:53:30.645 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:53:30.718 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records in 2022-03-08T22:53:30.718 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2022-03-08T22:53:30.718 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135603 s, 773 MB/s 2022-03-08T22:53:30.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:53:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records in 2022-03-08T22:53:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2022-03-08T22:53:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136003 s, 771 MB/s 2022-03-08T22:53:30.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:53:30.772 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records in 2022-03-08T22:53:30.772 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2022-03-08T22:53:30.773 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131387 s, 798 MB/s 2022-03-08T22:53:30.791 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T22:53:30.791 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi042-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi042 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi042-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi042 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi042-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi042 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi042-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi042 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi153-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi153 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi153-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi153 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi153-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi153 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi153-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi153 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi178-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi178 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi178-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi178 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi178-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi178 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi178-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi178 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T22:53:30.792 DEBUG:teuthology.orchestra.run.smithi042:> kubectl create -f - 2022-03-08T22:53:31.313 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi042-lv-1 created 2022-03-08T22:53:31.325 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi042-lv-2 created 2022-03-08T22:53:31.346 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi042-lv-3 created 2022-03-08T22:53:31.359 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi042-lv-4 created 2022-03-08T22:53:31.477 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi153-lv-1 created 2022-03-08T22:53:31.480 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi153-lv-2 created 2022-03-08T22:53:31.534 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi153-lv-3 created 2022-03-08T22:53:31.651 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi153-lv-4 created 2022-03-08T22:53:31.679 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi178-lv-1 created 2022-03-08T22:53:31.721 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi178-lv-2 created 2022-03-08T22:53:31.725 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi178-lv-3 created 2022-03-08T22:53:31.775 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi178-lv-4 created 2022-03-08T22:53:31.801 INFO:teuthology.orchestra.run.smithi042.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T22:53:31.809 DEBUG:teuthology.orchestra.run.smithi042:> kubectl taint node smithi042 node-role.kubernetes.io/master- || true 2022-03-08T22:53:31.953 INFO:teuthology.orchestra.run.smithi042.stdout:node/smithi042 untainted 2022-03-08T22:53:31.955 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T22:53:31.956 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T22:53:31.968 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T22:53:31.969 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:31.969 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:31.976 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:31.976 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:53:31.981 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:53:31.982 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:31.982 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T22:53:31.982 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:31.983 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2022-03-08 22:53:30.358043959 +0000 2022-03-08T22:53:31.983 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2022-03-08 22:53:30.358043959 +0000 2022-03-08T22:53:31.983 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2022-03-08 22:53:30.358043959 +0000 2022-03-08T22:53:31.983 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2022-03-08T22:53:31.990 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2022-03-08T22:53:31.991 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2022-03-08T22:53:31.991 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000244083 s, 2.1 MB/s 2022-03-08T22:53:31.995 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:53:32.000 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:53:32.001 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.001 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-08T22:53:32.001 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.001 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2022-03-08 22:53:30.382043959 +0000 2022-03-08T22:53:32.002 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2022-03-08 22:53:30.382043959 +0000 2022-03-08T22:53:32.002 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2022-03-08 22:53:30.382043959 +0000 2022-03-08T22:53:32.002 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2022-03-08T22:53:32.007 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2022-03-08T22:53:32.008 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2022-03-08T22:53:32.008 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000189547 s, 2.7 MB/s 2022-03-08T22:53:32.011 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:53:32.061 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:53:32.061 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.061 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T22:53:32.062 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.062 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2022-03-08 22:53:30.406043958 +0000 2022-03-08T22:53:32.062 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2022-03-08 22:53:30.406043958 +0000 2022-03-08T22:53:32.062 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2022-03-08 22:53:30.406043958 +0000 2022-03-08T22:53:32.063 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2022-03-08T22:53:32.070 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2022-03-08T22:53:32.070 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2022-03-08T22:53:32.070 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000281663 s, 1.8 MB/s 2022-03-08T22:53:32.073 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:53:32.079 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:53:32.079 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.079 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-08T22:53:32.079 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.079 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2022-03-08 22:53:30.430043958 +0000 2022-03-08T22:53:32.080 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2022-03-08 22:53:30.430043958 +0000 2022-03-08T22:53:32.080 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2022-03-08 22:53:30.430043958 +0000 2022-03-08T22:53:32.080 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2022-03-08T22:53:32.086 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2022-03-08T22:53:32.086 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2022-03-08T22:53:32.086 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000345864 s, 1.5 MB/s 2022-03-08T22:53:32.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:53:32.165 INFO:teuthology.orchestra.run.smithi042.stdout:loop 2022-03-08T22:53:32.166 INFO:tasks.nvme_loop:Connecting nvme_loop smithi042:/dev/vg_nvme/lv_1... 2022-03-08T22:53:32.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:53:32.186 INFO:teuthology.orchestra.run.smithi042.stdout:1 2022-03-08T22:53:32.214 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:53:32.235 INFO:tasks.nvme_loop:Connecting nvme_loop smithi042:/dev/vg_nvme/lv_2... 2022-03-08T22:53:32.236 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:53:32.253 INFO:teuthology.orchestra.run.smithi042.stdout:1 2022-03-08T22:53:32.276 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:53:32.296 INFO:tasks.nvme_loop:Connecting nvme_loop smithi042:/dev/vg_nvme/lv_3... 2022-03-08T22:53:32.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:53:32.315 INFO:teuthology.orchestra.run.smithi042.stdout:1 2022-03-08T22:53:32.338 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:53:32.363 INFO:tasks.nvme_loop:Connecting nvme_loop smithi042:/dev/vg_nvme/lv_4... 2022-03-08T22:53:32.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:53:32.385 INFO:teuthology.orchestra.run.smithi042.stdout:1 2022-03-08T22:53:32.414 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:53:32.437 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:32.438 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:32.445 DEBUG:teuthology.orchestra.run.smithi042:> sudo nvme list 2022-03-08T22:53:32.500 INFO:teuthology.orchestra.run.smithi042.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:53:32.500 INFO:teuthology.orchestra.run.smithi042.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:53:32.500 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/nvme0n1 PHMB75130021480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-08T22:53:32.501 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/nvme1n1 9d83617eea881749 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:32.501 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/nvme2n1 094aece474b74c56 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:32.501 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/nvme3n1 13935fd7bda24abe Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:32.501 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/nvme4n1 808a1c774bec9cf3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:32.502 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:53:32.503 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:32.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/scratch_devs 2022-03-08T22:53:32.515 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:32.516 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:32.524 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:32.524 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:53:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:53:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.572 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T22:53:32.572 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.573 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2022-03-08 22:53:30.468245190 +0000 2022-03-08T22:53:32.573 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2022-03-08 22:53:30.468245190 +0000 2022-03-08T22:53:32.573 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2022-03-08 22:53:30.468245190 +0000 2022-03-08T22:53:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2022-03-08T22:53:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2022-03-08T22:53:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2022-03-08T22:53:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000598294 s, 856 kB/s 2022-03-08T22:53:32.592 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:53:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:53:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.599 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-08T22:53:32.599 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.599 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2022-03-08 22:53:30.496245189 +0000 2022-03-08T22:53:32.599 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2022-03-08 22:53:30.496245189 +0000 2022-03-08T22:53:32.600 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2022-03-08 22:53:30.496245189 +0000 2022-03-08T22:53:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2022-03-08T22:53:32.609 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2022-03-08T22:53:32.609 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2022-03-08T22:53:32.610 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000606211 s, 845 kB/s 2022-03-08T22:53:32.613 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:53:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:53:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.620 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T22:53:32.621 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.621 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2022-03-08 22:53:30.580245188 +0000 2022-03-08T22:53:32.621 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2022-03-08 22:53:30.580245188 +0000 2022-03-08T22:53:32.621 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2022-03-08 22:53:30.580245188 +0000 2022-03-08T22:53:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2022-03-08T22:53:32.627 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2022-03-08T22:53:32.627 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2022-03-08T22:53:32.627 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000479515 s, 1.1 MB/s 2022-03-08T22:53:32.631 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:53:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:53:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:32.680 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-08T22:53:32.680 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:32.681 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2022-03-08 22:53:30.612245187 +0000 2022-03-08T22:53:32.681 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2022-03-08 22:53:30.612245187 +0000 2022-03-08T22:53:32.681 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2022-03-08 22:53:30.612245187 +0000 2022-03-08T22:53:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2022-03-08T22:53:32.689 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2022-03-08T22:53:32.689 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2022-03-08T22:53:32.689 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000573291 s, 893 kB/s 2022-03-08T22:53:32.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:53:32.772 INFO:teuthology.orchestra.run.smithi153.stdout:loop 2022-03-08T22:53:32.773 INFO:tasks.nvme_loop:Connecting nvme_loop smithi153:/dev/vg_nvme/lv_1... 2022-03-08T22:53:32.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:53:32.793 INFO:teuthology.orchestra.run.smithi153.stdout:1 2022-03-08T22:53:32.816 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:53:32.838 INFO:tasks.nvme_loop:Connecting nvme_loop smithi153:/dev/vg_nvme/lv_2... 2022-03-08T22:53:32.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:53:32.860 INFO:teuthology.orchestra.run.smithi153.stdout:1 2022-03-08T22:53:32.885 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:53:32.903 INFO:tasks.nvme_loop:Connecting nvme_loop smithi153:/dev/vg_nvme/lv_3... 2022-03-08T22:53:32.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:53:32.965 INFO:teuthology.orchestra.run.smithi153.stdout:1 2022-03-08T22:53:32.986 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:53:33.009 INFO:tasks.nvme_loop:Connecting nvme_loop smithi153:/dev/vg_nvme/lv_4... 2022-03-08T22:53:33.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:53:33.030 INFO:teuthology.orchestra.run.smithi153.stdout:1 2022-03-08T22:53:33.052 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:53:33.075 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:33.075 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:33.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo nvme list 2022-03-08T22:53:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:53:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:53:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/nvme0n1 CVFT623300MV400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV10171 2022-03-08T22:53:33.141 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/nvme1n1 638549c5cef19d84 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.141 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/nvme2n1 ae418b07654861a5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.141 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/nvme3n1 522ce432c2d4dc17 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.141 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/nvme4n1 f61ad758c31b21f5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.142 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:53:33.143 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:53:33.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/scratch_devs 2022-03-08T22:53:33.157 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:33.157 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:33.164 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:53:33.164 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:53:33.169 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:53:33.170 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:33.170 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 495 Links: 1 2022-03-08T22:53:33.170 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:33.170 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2022-03-08 22:53:30.648592171 +0000 2022-03-08T22:53:33.171 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2022-03-08 22:53:30.644592171 +0000 2022-03-08T22:53:33.171 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2022-03-08 22:53:30.644592171 +0000 2022-03-08T22:53:33.171 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2022-03-08T22:53:33.176 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2022-03-08T22:53:33.176 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2022-03-08T22:53:33.177 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000205581 s, 2.5 MB/s 2022-03-08T22:53:33.179 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:53:33.184 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:53:33.185 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:33.185 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-03-08T22:53:33.185 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:33.185 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2022-03-08 22:53:30.720592170 +0000 2022-03-08T22:53:33.186 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2022-03-08 22:53:30.720592170 +0000 2022-03-08T22:53:33.186 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2022-03-08 22:53:30.720592170 +0000 2022-03-08T22:53:33.186 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2022-03-08T22:53:33.192 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2022-03-08T22:53:33.192 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2022-03-08T22:53:33.192 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.0001735 s, 3.0 MB/s 2022-03-08T22:53:33.195 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:53:33.245 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:53:33.245 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:33.246 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-03-08T22:53:33.246 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:33.246 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2022-03-08 22:53:30.748592169 +0000 2022-03-08T22:53:33.246 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2022-03-08 22:53:30.748592169 +0000 2022-03-08T22:53:33.247 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2022-03-08 22:53:30.748592169 +0000 2022-03-08T22:53:33.247 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2022-03-08T22:53:33.255 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2022-03-08T22:53:33.256 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2022-03-08T22:53:33.256 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000262954 s, 1.9 MB/s 2022-03-08T22:53:33.259 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:53:33.265 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:53:33.266 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:53:33.266 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 515 Links: 1 2022-03-08T22:53:33.266 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:53:33.267 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2022-03-08 22:53:30.776592168 +0000 2022-03-08T22:53:33.267 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2022-03-08 22:53:30.776592168 +0000 2022-03-08T22:53:33.267 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2022-03-08 22:53:30.776592168 +0000 2022-03-08T22:53:33.267 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2022-03-08T22:53:33.274 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2022-03-08T22:53:33.274 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2022-03-08T22:53:33.274 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000322717 s, 1.6 MB/s 2022-03-08T22:53:33.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:53:33.460 INFO:teuthology.orchestra.run.smithi178.stdout:loop 2022-03-08T22:53:33.461 INFO:tasks.nvme_loop:Connecting nvme_loop smithi178:/dev/vg_nvme/lv_1... 2022-03-08T22:53:33.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:53:33.483 INFO:teuthology.orchestra.run.smithi178.stdout:1 2022-03-08T22:53:33.512 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:53:33.532 INFO:tasks.nvme_loop:Connecting nvme_loop smithi178:/dev/vg_nvme/lv_2... 2022-03-08T22:53:33.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:53:33.551 INFO:teuthology.orchestra.run.smithi178.stdout:1 2022-03-08T22:53:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:53:33.594 INFO:tasks.nvme_loop:Connecting nvme_loop smithi178:/dev/vg_nvme/lv_3... 2022-03-08T22:53:33.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:53:33.612 INFO:teuthology.orchestra.run.smithi178.stdout:1 2022-03-08T22:53:33.632 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:53:33.650 INFO:tasks.nvme_loop:Connecting nvme_loop smithi178:/dev/vg_nvme/lv_4... 2022-03-08T22:53:33.651 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:53:33.667 INFO:teuthology.orchestra.run.smithi178.stdout:1 2022-03-08T22:53:33.688 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:53:33.707 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:33.708 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:33.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo nvme list 2022-03-08T22:53:33.766 INFO:teuthology.orchestra.run.smithi178.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:53:33.766 INFO:teuthology.orchestra.run.smithi178.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:53:33.766 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/nvme0n1 PHMB75130029480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-08T22:53:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/nvme1n1 074ca5075bfa085c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/nvme2n1 80e219152887dc24 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/nvme3n1 bf170555df358773 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:/dev/nvme4n1 969cedbbd58a8a29 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:53:33.768 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:53:33.769 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:53:33.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/scratch_devs 2022-03-08T22:53:33.870 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T22:53:33.944 INFO:tasks.rook:Rook start 2022-03-08T22:53:33.944 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4', 'rook_image': 'rook/ceph:master'} 2022-03-08T22:53:33.945 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:53:33.945 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-08T22:53:33.945 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-08T22:53:33.955 INFO:teuthology.orchestra.run.smithi042.stderr:Cloning into 'rook'... 2022-03-08T22:53:36.499 INFO:tasks.rook:/home/teuthworker 2022-03-08T22:53:36.499 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T22:53:36.553 INFO:teuthology.orchestra.run.smithi042.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T22:53:36.553 INFO:teuthology.orchestra.run.smithi042.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T22:53:36.553 INFO:teuthology.orchestra.run.smithi042.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T22:53:36.554 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:53:36.554 INFO:teuthology.orchestra.run.smithi042.stdout: inet6 ::1/128 scope host 2022-03-08T22:53:36.554 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:53:36.554 INFO:teuthology.orchestra.run.smithi042.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:53:36.555 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:6b:fd:0e brd ff:ff:ff:ff:ff:ff 2022-03-08T22:53:36.555 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp7s0f0 2022-03-08T22:53:36.555 INFO:teuthology.orchestra.run.smithi042.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:53:36.555 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:6b:fd:0f brd ff:ff:ff:ff:ff:ff 2022-03-08T22:53:36.556 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp7s0f1 2022-03-08T22:53:36.556 INFO:teuthology.orchestra.run.smithi042.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T22:53:36.556 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:bd:14:40 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:53:36.556 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp1s0f0 2022-03-08T22:53:36.557 INFO:teuthology.orchestra.run.smithi042.stdout: inet 172.21.15.42/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-08T22:53:36.557 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft 42848sec preferred_lft 42848sec 2022-03-08T22:53:36.557 INFO:teuthology.orchestra.run.smithi042.stdout: inet6 fe80::ec4:7aff:febd:1440/64 scope link 2022-03-08T22:53:36.557 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:53:36.558 INFO:teuthology.orchestra.run.smithi042.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:53:36.558 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:bd:14:41 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:53:36.558 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp1s0f1 2022-03-08T22:53:36.558 INFO:teuthology.orchestra.run.smithi042.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T22:53:36.558 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 02:42:fc:e4:f4:77 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:53:36.559 INFO:teuthology.orchestra.run.smithi042.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T22:53:36.559 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:53:36.560 DEBUG:teuthology.orchestra.run.smithi042:> which systemctl 2022-03-08T22:53:36.565 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/systemctl 2022-03-08T22:53:36.565 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T22:53:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T22:53:36.571 DEBUG:teuthology.orchestra.run.smithi042:> true 2022-03-08T22:53:36.647 DEBUG:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2022-03-08T22:53:36.696 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2022-03-08T22:53:36.702 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:53:36.702 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T22:53:36.711 DEBUG:teuthology.orchestra.remote:smithi042:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T22:53:36.725 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:36.725 DEBUG:teuthology.orchestra.run.smithi042:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-08T22:53:36.734 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-08T22:53:36.774 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:53:36.774 DEBUG:teuthology.orchestra.run.smithi042:> dd of=operator.yaml 2022-03-08T22:53:36.787 INFO:tasks.rook:Deploying operator 2022-03-08T22:53:36.796 DEBUG:teuthology.orchestra.remote:smithi042:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T22:53:36.818 DEBUG:teuthology.orchestra.remote:smithi042:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T22:53:36.827 DEBUG:teuthology.orchestra.run.smithi042:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-08T22:53:37.112 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T22:53:37.119 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-08T22:53:37.165 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-08T22:53:37.269 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T22:53:37.385 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T22:53:37.482 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T22:53:37.531 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T22:53:37.579 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-08T22:53:37.682 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T22:53:37.729 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T22:53:37.850 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T22:53:37.879 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T22:53:37.901 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T22:53:38.005 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T22:53:38.067 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T22:53:38.190 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T22:53:38.218 INFO:teuthology.orchestra.run.smithi042.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T22:53:38.390 INFO:teuthology.orchestra.run.smithi042.stdout:namespace/rook-ceph created 2022-03-08T22:53:38.402 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:53:38.424 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T22:53:38.433 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T22:53:38.457 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:53:38.483 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T22:53:38.551 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:53:38.576 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:53:38.600 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:53:38.626 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:53:38.710 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:53:38.713 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:53:38.743 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:53:38.755 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:53:38.806 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T22:53:38.867 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:53:38.870 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T22:53:38.874 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:53:38.883 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:53:38.891 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:53:38.919 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:53:38.925 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:53:38.935 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T22:53:38.941 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T22:53:38.949 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T22:53:38.958 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T22:53:39.008 INFO:teuthology.orchestra.run.smithi042.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T22:53:39.012 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:53:39.013 INFO:teuthology.orchestra.run.smithi042.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T22:53:39.016 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T22:53:39.024 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:53:39.033 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T22:53:39.061 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:53:39.066 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:53:39.075 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:53:39.083 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:53:39.091 INFO:teuthology.orchestra.run.smithi042.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:53:39.162 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T22:53:39.165 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-08T22:53:39.183 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T22:53:39.195 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:53:39.208 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:53:39.220 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T22:53:39.233 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T22:53:39.312 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:53:39.316 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T22:53:39.342 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:53:39.350 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:53:39.358 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T22:53:39.366 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:53:39.389 INFO:teuthology.orchestra.run.smithi042.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:53:39.399 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T22:53:39.458 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T22:53:39.461 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T22:53:39.487 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T22:53:39.515 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-ceph-system created 2022-03-08T22:53:39.608 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T22:53:39.631 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T22:53:39.681 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T22:53:39.707 INFO:teuthology.orchestra.run.smithi042.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T22:53:39.800 INFO:teuthology.orchestra.run.smithi042.stdout:configmap/rook-ceph-operator-config created 2022-03-08T22:53:39.834 INFO:teuthology.orchestra.run.smithi042.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T22:53:39.839 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:No resources found in rook-ceph namespace. 2022-03-08T22:53:50.007 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:50.069 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:50.069 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-operator-75dd789779-gctqr 0/1 ContainerCreating 0 11s 2022-03-08T22:54:00.073 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:54:00.138 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:00.139 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-operator-75dd789779-gctqr 0/1 ContainerCreating 0 21s 2022-03-08T22:54:10.142 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:54:10.207 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:10.207 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-operator-75dd789779-gctqr 0/1 ContainerCreating 0 31s 2022-03-08T22:54:20.210 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:54:20.280 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:20.281 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-operator-75dd789779-gctqr 1/1 Running 0 41s 2022-03-08T22:54:20.283 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-gctqr 2022-03-08T22:54:20.291 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:54:20.291 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:54:20.338 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T22:54:20.338 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:54:20.346 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T22:54:20.347 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:54:20.355 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T22:54:20.356 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T22:54:20.356 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T22:54:20.356 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T22:54:20.357 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T22:54:20.357 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T22:54:20.357 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T22:54:20.358 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T22:54:20.360 DEBUG:teuthology.orchestra.run.smithi042:> kubectl create -f - 2022-03-08T22:54:20.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.594368 I | rookcmd: starting Rook v1.8.0-alpha.0.381.g235781e28 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T22:54:20.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.594553 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-08T22:54:20.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.594556 I | cephcmd: starting Rook-Ceph operator 2022-03-08T22:54:20.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.772612 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-08T22:54:20.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.793489 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T22:54:20.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.793514 I | operator: watching all namespaces for Ceph CRs 2022-03-08T22:54:20.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.793606 I | operator: setting up schemes 2022-03-08T22:54:20.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:18.796906 I | operator: setting up the controller-runtime manager 2022-03-08T22:54:20.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.001246 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T22:54:20.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004073 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T22:54:20.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004127 I | ceph-cluster-controller: successfully started 2022-03-08T22:54:20.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004221 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T22:54:20.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004259 I | ceph-crashcollector-controller: successfully started 2022-03-08T22:54:20.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004296 I | ceph-block-pool-controller: successfully started 2022-03-08T22:54:20.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004354 I | ceph-object-store-user-controller: successfully started 2022-03-08T22:54:20.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004396 I | ceph-object-realm-controller: successfully started 2022-03-08T22:54:20.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004425 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T22:54:20.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004454 I | ceph-object-zone-controller: successfully started 2022-03-08T22:54:20.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004599 I | ceph-object-controller: successfully started 2022-03-08T22:54:20.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004639 I | ceph-file-controller: successfully started 2022-03-08T22:54:20.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004689 I | ceph-nfs-controller: successfully started 2022-03-08T22:54:20.383 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004738 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T22:54:20.383 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004782 I | ceph-client-controller: successfully started 2022-03-08T22:54:20.383 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004808 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T22:54:20.383 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004864 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T22:54:20.384 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004892 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T22:54:20.384 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004908 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T22:54:20.384 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004938 I | ceph-bucket-topic: successfully started 2022-03-08T22:54:20.384 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004956 I | ceph-bucket-notification: successfully started 2022-03-08T22:54:20.384 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004970 I | ceph-bucket-notification: successfully started 2022-03-08T22:54:20.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.004986 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T22:54:20.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.006108 I | operator: starting the controller-runtime manager 2022-03-08T22:54:20.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.109487 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T22:54:20.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.109531 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T22:54:20.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.109553 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T22:54:20.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.111995 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-08T22:54:20.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.112043 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-08T22:54:20.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.123000 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-08T22:54:20.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.123082 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T22:54:20.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.130190 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-08T22:54:20.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.130213 I | ceph-csi: CSI CephFS driver disabled 2022-03-08T22:54:20.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.130233 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-08T22:54:20.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.132349 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-08T22:54:20.388 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:19.138753 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-08T22:54:21.237 INFO:teuthology.orchestra.run.smithi042.stdout:configmap/rook-config-override created 2022-03-08T22:54:21.242 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-08T22:54:21.243 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T22:54:21.243 DEBUG:teuthology.orchestra.run.smithi042:> dd of=cluster.yaml 2022-03-08T22:54:21.250 DEBUG:teuthology.orchestra.run.smithi042:> kubectl create -f cluster.yaml 2022-03-08T22:54:23.449 INFO:teuthology.orchestra.run.smithi042.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T22:54:23.456 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.458584 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T22:54:23.463 DEBUG:teuthology.orchestra.remote:smithi042:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T22:54:23.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.480383 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:54:23.479 DEBUG:teuthology.orchestra.run.smithi042:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-08T22:54:23.528 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.531234 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-08T22:54:23.542 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.545872 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T22:54:23.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.551696 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:54:23.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625389 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T22:54:23.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625417 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T22:54:23.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625426 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T22:54:23.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625434 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T22:54:23.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625441 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T22:54:23.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625450 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T22:54:23.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625459 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T22:54:23.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625468 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T22:54:23.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625482 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T22:54:23.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625496 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T22:54:23.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625509 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T22:54:23.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625525 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T22:54:23.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625539 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T22:54:23.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625546 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T22:54:23.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625554 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T22:54:23.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625564 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T22:54:23.629 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625700 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:23.629 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:54:23.625725 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:23.756 INFO:teuthology.orchestra.run.smithi042.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T22:54:23.761 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T22:54:23.761 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:23.887 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:23.887 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 Pending 0 0s 2022-03-08T22:54:28.890 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:28.956 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:28.957 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 5s 2022-03-08T22:54:33.959 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:34.023 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:34.023 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 11s 2022-03-08T22:54:39.025 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:39.089 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:39.090 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 16s 2022-03-08T22:54:44.093 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:44.154 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:44.155 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 21s 2022-03-08T22:54:49.156 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:49.228 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:49.229 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 26s 2022-03-08T22:54:54.231 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:54.300 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:54.301 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 31s 2022-03-08T22:54:59.303 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:59.364 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:59.364 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 36s 2022-03-08T22:55:04.368 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:04.437 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:04.438 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 41s 2022-03-08T22:55:09.439 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:09.509 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:09.509 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 46s 2022-03-08T22:55:14.512 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:14.577 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:14.577 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 51s 2022-03-08T22:55:19.580 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:19.646 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:19.646 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 56s 2022-03-08T22:55:24.649 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:24.722 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:24.723 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 61s 2022-03-08T22:55:28.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.862604 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:55:28.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.862634 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:55:28.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.862649 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:55:28.858 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.865329 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:55:28.925 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.933930 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 22:55:28.902973302 +0000 UTC m=+70.330276389 LastTransitionTime:2022-03-08 22:55:28.902973082 +0000 UTC m=+70.330276196}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:55:28.938 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:28.947324 I | op-mon: start running mons 2022-03-08T22:55:29.064 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:29.072897 I | op-mon: creating mon secrets for a new cluster 2022-03-08T22:55:29.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:29.090149 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T22:55:29.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:29.093288 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T22:55:29.726 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:29.755 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:29.763822 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:55:29.755 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:29.764035 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:55:29.793 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:29.794 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 66s 2022-03-08T22:55:31.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:31.363881 I | op-mon: targeting the mon count 3 2022-03-08T22:55:31.371 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:31.380832 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T22:55:31.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:31.395550 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-08T22:55:31.454 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:31.463684 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-08T22:55:32.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.164879 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi153 2022-03-08T22:55:32.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.164915 I | op-mon: mon a assigned to node smithi153 2022-03-08T22:55:32.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.364183 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi178 2022-03-08T22:55:32.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.364212 I | op-mon: mon b assigned to node smithi178 2022-03-08T22:55:32.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.563731 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi042 2022-03-08T22:55:32.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.563764 I | op-mon: mon c assigned to node smithi042 2022-03-08T22:55:32.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.575638 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T22:55:32.572 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.581568 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-08T22:55:32.746 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.755578 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-08T22:55:32.753 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.762157 I | op-mon: creating mon a 2022-03-08T22:55:32.943 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:32.946697 I | op-mon: mon "a" endpoint is [v2:10.100.208.21:3300,v1:10.100.208.21:6789] 2022-03-08T22:55:33.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:33.370965 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:55:33.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:33.371079 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:55:33.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:33.371290 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789"]}] data:a=10.100.208.21:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:-1] 2022-03-08T22:55:34.164 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.173581 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:55:34.165 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.173731 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:55:34.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.362623 I | op-mon: parsing mon endpoints: a=10.100.208.21:6789 2022-03-08T22:55:34.354 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.362655 I | op-mon: updating obsolete maxMonID -1 to actual value 0 2022-03-08T22:55:34.354 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.362705 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T22:55:34.354 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.362715 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:34.357 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.364977 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T22:55:34.357 INFO:tasks.rook.operator.smithi042.stdout:I0308 22:55:34.365062 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:34.770 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:34.779148 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T22:55:34.796 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:34.875 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:34.876 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 71s 2022-03-08T22:55:35.056 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:35.065722 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T22:55:35.753 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:35.762733 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789"]}] data:a=10.100.208.21:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:0] 2022-03-08T22:55:35.754 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:35.762775 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:35.968 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:35.977599 I | op-mon: mon a is not yet running 2022-03-08T22:55:35.969 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:35.977632 I | op-mon: mons running: [] 2022-03-08T22:55:39.880 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:39.949 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:39.949 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 76s 2022-03-08T22:55:44.954 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:45.025 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:45.025 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 82s 2022-03-08T22:55:50.033 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:50.098 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:50.098 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 87s 2022-03-08T22:55:55.102 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:55.162 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:55.163 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 92s 2022-03-08T22:55:56.969 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:56.978097 I | op-mon: mons running: [a] 2022-03-08T22:55:57.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.206119 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:57.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.206155 I | op-mon: mons created: 1 2022-03-08T22:55:57.426 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.435675 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:57.437 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.447072 I | op-mon: mons running: [a] 2022-03-08T22:55:57.665 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.674889 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:57.665 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.674928 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:57.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.984650 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:57.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:57.984677 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:58.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.278437 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:58.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.278474 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:58.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.550268 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:58.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.550305 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:58.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.869626 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:58.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:58.869655 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:55:59.101 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:59.110767 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:55:59.101 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:55:59.110800 I | op-mon: creating mon b 2022-03-08T22:56:00.167 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:00.602 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:00.603 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 97s 2022-03-08T22:56:01.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:01.150551 I | op-mon: mon "a" endpoint is [v2:10.100.208.21:3300,v1:10.100.208.21:6789] 2022-03-08T22:56:01.561 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:01.571084 I | op-mon: mon "b" endpoint is [v2:10.110.240.83:3300,v1:10.110.240.83:6789] 2022-03-08T22:56:02.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.215232 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.240.83:6789","10.100.208.21:6789"]}] data:a=10.100.208.21:6789,b=10.110.240.83:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:0] 2022-03-08T22:56:02.502 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.512435 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:02.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.512722 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:02.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.565384 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:56:02.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.572389 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:56:02.563 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.572408 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T22:56:02.571 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.581365 I | op-mon: mons running: [a b] 2022-03-08T22:56:02.853 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:02.861982 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:03.167 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:03.177614 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-08T22:56:04.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:04.157598 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789","10.110.240.83:6789"]}] data:b=10.110.240.83:6789,a=10.100.208.21:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:1] 2022-03-08T22:56:04.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:04.157638 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T22:56:04.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:04.687358 I | op-mon: mons running: [a b] 2022-03-08T22:56:04.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:04.909143 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:04.900 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:04.909176 I | op-mon: mons created: 2 2022-03-08T22:56:05.174 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:05.184086 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T22:56:05.605 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:05.615 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:05.625300 I | op-mon: mons running: [a b] 2022-03-08T22:56:05.702 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:05.703 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 102s 2022-03-08T22:56:05.831 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:05.841487 W | op-mon: failed to find initial monitor b in mon map 2022-03-08T22:56:05.832 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:05.841515 W | op-mon: monitor b is not in quorum list 2022-03-08T22:56:10.706 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:10.785 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:10.785 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 107s 2022-03-08T22:56:10.864 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:10.873875 I | op-mon: mon b is not yet running 2022-03-08T22:56:10.864 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:10.873901 I | op-mon: mons running: [a] 2022-03-08T22:56:15.788 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:15.857 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:15.857 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 112s 2022-03-08T22:56:15.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:15.901033 I | op-mon: mon b is not yet running 2022-03-08T22:56:15.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:15.901057 I | op-mon: mons running: [a] 2022-03-08T22:56:16.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452498 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T22:56:16.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452540 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T22:56:16.444 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452550 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:56:16.444 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452558 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:56:16.444 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452566 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:56:16.446 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452573 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:56:16.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452580 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:56:16.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452588 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:56:16.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452596 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T22:56:16.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452603 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T22:56:16.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452611 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:56:16.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452618 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:56:16.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452626 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:56:16.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452633 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:56:16.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452640 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T22:56:16.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452648 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T22:56:16.449 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452655 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T22:56:16.449 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452662 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T22:56:16.449 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452669 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T22:56:16.449 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452677 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:56:16.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452685 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:56:16.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452692 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T22:56:16.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452699 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T22:56:16.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452709 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T22:56:16.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.452715 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T22:56:16.455 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.464583 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T22:56:16.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472837 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:56:16.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472861 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472867 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:56:16.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472871 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472876 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:56:16.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472881 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.472885 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T22:56:16.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.507930 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:56:16.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.507953 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.499 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.507958 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:56:16.602 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.612292 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T22:56:16.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.686827 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:56:16.678 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.686846 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.678 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.686851 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T22:56:16.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.804581 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:56:16.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.804612 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:56:16.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.804619 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:56:16.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:16.880274 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T22:56:17.005 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:17.014689 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:56:17.105 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:17.115161 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T22:56:17.105 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:17.115193 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:56:17.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:17.151769 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T22:56:20.861 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:20.915 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:20.925332 I | op-mon: mon b is not yet running 2022-03-08T22:56:20.915 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:20.925366 I | op-mon: mons running: [a] 2022-03-08T22:56:20.926 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:20.927 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 117s 2022-03-08T22:56:25.929 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:25.996 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:25.996 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 2m2s 2022-03-08T22:56:26.007 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:26.001753 I | op-mon: mon b is not yet running 2022-03-08T22:56:26.007 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:26.001776 I | op-mon: mons running: [a] 2022-03-08T22:56:30.998 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:31.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:31.033724 I | op-mon: mon b is not yet running 2022-03-08T22:56:31.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:31.033761 I | op-mon: mons running: [a] 2022-03-08T22:56:31.082 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:31.082 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 2m8s 2022-03-08T22:56:36.052 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:36.062379 I | op-mon: mon b is not yet running 2022-03-08T22:56:36.052 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:36.062437 I | op-mon: mons running: [a] 2022-03-08T22:56:36.085 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:36.178 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:36.178 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 2m13s 2022-03-08T22:56:41.182 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:41.648 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:41.649 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 0/1 ContainerCreating 0 2m18s 2022-03-08T22:56:41.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:41.660770 I | op-mon: mons running: [a b] 2022-03-08T22:56:44.870 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:44.880877 I | op-mon: Monitors in quorum: [a b] 2022-03-08T22:56:44.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:44.880914 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:45.086 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.096404 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:45.086 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.096438 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:45.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.594204 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:45.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.594316 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:45.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.857388 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:45.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:45.857424 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:46.110 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.120278 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:46.110 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.120306 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:56:46.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.365399 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:56:46.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.365425 I | op-mon: creating mon c 2022-03-08T22:56:46.520 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.531088 I | op-mon: mon "a" endpoint is [v2:10.100.208.21:3300,v1:10.100.208.21:6789] 2022-03-08T22:56:46.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.636808 I | op-mon: mon "b" endpoint is [v2:10.110.240.83:3300,v1:10.110.240.83:6789] 2022-03-08T22:56:46.652 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:56:46.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.673632 I | op-mon: mon "c" endpoint is [v2:10.111.142.38:3300,v1:10.111.142.38:6789] 2022-03-08T22:56:46.737 INFO:teuthology.orchestra.run.smithi042.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:56:46.737 INFO:teuthology.orchestra.run.smithi042.stdout:rook-ceph-tools-d6d7c985c-tbfns 1/1 Running 0 2m23s 2022-03-08T22:56:46.739 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T22:56:46.739 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:56:46.760 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:46.770787 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.111.142.38:6789","10.100.208.21:6789","10.110.240.83:6789"]}] data:a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:1] 2022-03-08T22:56:47.234 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:56:47.235 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:56:47.235 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:56:47.235 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:56:47.236 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:56:47.236 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:56:47.236 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:56:47.236 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:56:47.236 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:56:47.237 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:56:47.237 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:56:47.237 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:56:47.237 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:56:47.238 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:56:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:56:47.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.391351 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:47.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.391636 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:47.783 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.793488 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-08T22:56:47.789 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.800564 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:56:47.798 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.809579 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:56:47.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.814495 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T22:56:47.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.825137 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T22:56:47.981 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:47.991604 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-08T22:56:49.042 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.053151 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789","10.110.240.83:6789","10.111.142.38:6789"]}] data:a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T22:56:49.043 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.053190 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:56:49.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.433736 I | op-mon: mon c is not yet running 2022-03-08T22:56:49.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.433770 I | op-mon: mons running: [a b] 2022-03-08T22:56:49.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.689776 I | op-mon: Monitors in quorum: [a b] 2022-03-08T22:56:49.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.689807 I | op-mon: mons created: 3 2022-03-08T22:56:49.898 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.908875 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:56:49.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.974924 I | op-mon: mon c is not yet running 2022-03-08T22:56:49.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:49.974955 I | op-mon: mons running: [a b] 2022-03-08T22:56:54.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:54.993523 I | op-mon: mon c is not yet running 2022-03-08T22:56:54.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:56:54.993549 I | op-mon: mons running: [a b] 2022-03-08T22:56:57.241 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:56:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:56:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:56:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:56:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:56:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:56:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:56:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:56:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:56:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:56:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:56:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:56:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:56:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:56:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:56:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:00.006 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:00.016740 I | op-mon: mon c is not yet running 2022-03-08T22:57:00.006 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:00.016777 I | op-mon: mons running: [a b] 2022-03-08T22:57:05.025 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:05.036387 I | op-mon: mon c is not yet running 2022-03-08T22:57:05.026 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:05.036429 I | op-mon: mons running: [a b] 2022-03-08T22:57:07.717 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:57:08.191 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:57:08.191 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:57:08.192 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:57:08.192 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:57:08.192 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:57:08.192 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:57:08.193 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:57:08.193 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:57:08.193 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:57:08.193 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:57:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:57:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:57:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:57:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:57:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:10.047 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:10.058149 I | op-mon: mon c is not yet running 2022-03-08T22:57:10.047 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:10.058187 I | op-mon: mons running: [a b] 2022-03-08T22:57:15.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:15.679218 I | op-mon: mon c is not yet running 2022-03-08T22:57:15.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:15.679253 I | op-mon: mons running: [a b] 2022-03-08T22:57:18.199 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:57:18.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:57:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:57:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:57:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:57:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:57:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:57:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:20.720 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:20.731577 I | op-mon: mon c is not yet running 2022-03-08T22:57:20.720 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:20.731606 I | op-mon: mons running: [a b] 2022-03-08T22:57:25.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:25.918878 I | op-mon: mon c is not yet running 2022-03-08T22:57:25.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:25.918922 I | op-mon: mons running: [a b] 2022-03-08T22:57:28.909 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:57:29.454 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:57:29.455 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:57:29.455 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:57:29.456 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:57:29.456 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:57:29.456 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:57:29.456 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:57:29.457 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:57:29.457 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:57:29.457 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:57:29.457 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:57:29.458 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:57:29.458 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:57:29.458 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:57:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:31.445 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:31.456369 I | op-mon: mon c is not yet running 2022-03-08T22:57:31.445 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:31.456407 I | op-mon: mons running: [a b] 2022-03-08T22:57:36.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:36.561385 I | op-mon: mon c is not yet running 2022-03-08T22:57:36.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:36.561423 I | op-mon: mons running: [a b] 2022-03-08T22:57:39.461 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:57:40.804 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:57:40.805 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:57:40.805 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:57:40.805 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:57:40.806 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:57:40.806 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:57:40.806 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:57:40.806 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:57:40.807 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:57:40.807 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:57:40.807 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:57:40.807 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:57:40.807 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:57:40.809 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:57:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:43.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:43.435573 I | op-mon: mon c is not yet running 2022-03-08T22:57:43.424 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:43.435611 I | op-mon: mons running: [a b] 2022-03-08T22:57:49.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:49.185194 I | op-mon: mon c is not yet running 2022-03-08T22:57:49.174 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:49.185234 I | op-mon: mons running: [a b] 2022-03-08T22:57:50.812 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:57:51.683 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:57:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:57:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:57:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:57:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:57:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:57:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:57:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:57:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:57:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:57:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:57:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:57:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:57:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:57:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:57:54.330 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:54.342143 I | op-mon: mon c is not yet running 2022-03-08T22:57:54.330 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:54.342180 I | op-mon: mons running: [a b] 2022-03-08T22:57:59.520 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:57:59.532158 I | op-mon: mons running: [a b c] 2022-03-08T22:58:01.690 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:58:02.915 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:58:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:58:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:58:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:58:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:58:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:58:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:58:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:58:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:58:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:58:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:58:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:58:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:58:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:58:03.017 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.029049 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T22:58:03.017 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.029076 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:58:03.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.241755 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:58:03.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.241790 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:58:03.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.462580 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:58:03.451 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.462616 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:58:03.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.675420 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:58:03.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.675472 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:58:03.878 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.890137 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:58:03.878 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:03.890155 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:58:04.113 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:04.124799 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:58:04.113 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:04.124838 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:58:04.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:04.474815 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:58:04.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:04.824061 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:58:05.159 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:05.171400 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:58:05.561 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:05.573590 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:58:05.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:05.573628 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:58:05.951 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:05.963313 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:58:06.189 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.201478 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:58:06.190 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.201509 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:58:06.190 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.201515 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:58:06.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.611723 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:58:06.618 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.630098 I | op-mgr: start running mgr 2022-03-08T22:58:06.618 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:06.630130 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:58:07.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:07.399006 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi178": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:58:12.922 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:58:13.360 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:13.361 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2022-03-08T22:58:13.361 INFO:teuthology.orchestra.run.smithi042.stderr:pg stat 2022-03-08T22:58:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:pg getmap 2022-03-08T22:58:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:58:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:58:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_pools_json 2022-03-08T22:58:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-pool [...] 2022-03-08T22:58:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-primary [] [...] 2022-03-08T22:58:13.364 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls-by-osd [] [...] 2022-03-08T22:58:13.364 INFO:teuthology.orchestra.run.smithi042.stderr:pg ls [] [...] 2022-03-08T22:58:13.364 INFO:teuthology.orchestra.run.smithi042.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:58:13.365 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2022-03-08T22:58:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 22 2022-03-08T22:58:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:58:23.371 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:58:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-08T22:58:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 2 2022-03-08T22:58:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-08T22:58:28.171 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.183650 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T22:58:28.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.187817 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:58:28.228 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.241282 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T22:58:28.229 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.241398 I | op-mgr: successful modules: balancer 2022-03-08T22:58:28.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.333364 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:58:28.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.333401 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T22:58:28.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.333416 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:58:28.325 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.337832 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:58:28.333 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.341088 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T22:58:28.333 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.341116 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:58:28.334 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.341125 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:58:28.736 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.748938 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:58:28.737 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.748971 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:58:28.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.762768 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:58:28.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.762801 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:58:28.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:28.762818 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:58:29.687 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:29.700089 I | op-mgr: successful modules: prometheus 2022-03-08T22:58:29.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:29.701282 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:58:29.694 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:29.707750 I | op-mgr: successful modules: dashboard 2022-03-08T22:58:34.286 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:58:35.195 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:35.207808 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi153": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:58:39.202 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:39.204 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-08T22:58:39.209 INFO:teuthology.orchestra.run.smithi042.stderr:command terminated with exit code 2 2022-03-08T22:58:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-08T22:58:39.320 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:39.332432 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:58:49.214 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status -f json 2022-03-08T22:58:49.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:49.853 INFO:teuthology.orchestra.run.smithi042.stdout:{"available": true, "backend": "rook"} 2022-03-08T22:58:49.875 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T22:58:49.876 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T22:58:50.679 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch apply osd --all-available-devices 2022-03-08T22:58:51.426 INFO:teuthology.orchestra.run.smithi042.stdout:['Success'] 2022-03-08T22:58:51.427 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.439781 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T22:58:51.427 INFO:tasks.rook.operator.smithi042.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T22:58:51.428 INFO:tasks.rook.operator.smithi042.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T22:58:51.428 INFO:tasks.rook.operator.smithi042.stdout:   ... // 3 identical fields 2022-03-08T22:58:51.428 INFO:tasks.rook.operator.smithi042.stdout:   Config: nil, 2022-03-08T22:58:51.435 INFO:tasks.rook.operator.smithi042.stdout:   Selection: {}, 2022-03-08T22:58:51.435 INFO:tasks.rook.operator.smithi042.stdout:-  StorageClassDeviceSets: nil, 2022-03-08T22:58:51.435 INFO:tasks.rook.operator.smithi042.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T22:58:51.436 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.436 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi042-lv-1", 2022-03-08T22:58:51.436 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.436 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.436 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.437 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.437 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi042-lv-2", 2022-03-08T22:58:51.437 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.437 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.438 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.438 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.438 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi042-lv-3", 2022-03-08T22:58:51.438 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.438 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.439 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.439 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.439 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi042-lv-4", 2022-03-08T22:58:51.439 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.440 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.440 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.440 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.440 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi153-lv-1", 2022-03-08T22:58:51.440 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.441 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.441 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.441 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.441 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi153-lv-2", 2022-03-08T22:58:51.442 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.442 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.442 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.442 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.442 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi153-lv-3", 2022-03-08T22:58:51.443 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.443 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.443 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.443 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.444 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi153-lv-4", 2022-03-08T22:58:51.444 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.444 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.444 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.444 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.445 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi178-lv-1", 2022-03-08T22:58:51.445 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.445 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.445 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.445 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.446 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi178-lv-2", 2022-03-08T22:58:51.446 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.446 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.446 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.447 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.447 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi178-lv-3", 2022-03-08T22:58:51.447 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.447 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.448 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.448 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T22:58:51.448 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi178-lv-4", 2022-03-08T22:58:51.448 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T22:58:51.448 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:58:51.449 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.449 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T22:58:51.449 INFO:tasks.rook.operator.smithi042.stdout:   }, 2022-03-08T22:58:51.450 INFO:tasks.rook.operator.smithi042.stdout:   Annotations: nil, 2022-03-08T22:58:51.450 INFO:tasks.rook.operator.smithi042.stdout:   Labels: nil, 2022-03-08T22:58:51.450 INFO:tasks.rook.operator.smithi042.stdout:   ... // 20 identical fields 2022-03-08T22:58:51.450 INFO:tasks.rook.operator.smithi042.stdout:  } 2022-03-08T22:58:51.451 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.439920 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T22:58:51.451 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.440212 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-08T22:58:51.451 INFO:tasks.rook.operator.smithi042.stdout:I0308 22:58:51.440460 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T22:58:51.451 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.440462 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-08T22:58:51.452 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.440485 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-08T22:58:51.452 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.440508 I | operator: successfully started the controller-runtime manager 2022-03-08T22:58:51.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.463316 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T22:58:51.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.463337 I | operator: watching all namespaces for Ceph CRs 2022-03-08T22:58:51.454 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.463368 I | operator: setting up schemes 2022-03-08T22:58:51.455 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-08T22:58:51.456 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:58:51.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.466864 I | operator: setting up the controller-runtime manager 2022-03-08T22:58:51.657 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.670703 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T22:58:51.662 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673649 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T22:58:51.662 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673689 I | ceph-cluster-controller: successfully started 2022-03-08T22:58:51.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673739 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T22:58:51.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673761 I | ceph-crashcollector-controller: successfully started 2022-03-08T22:58:51.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673779 I | ceph-block-pool-controller: successfully started 2022-03-08T22:58:51.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673808 I | ceph-object-store-user-controller: successfully started 2022-03-08T22:58:51.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673845 I | ceph-object-realm-controller: successfully started 2022-03-08T22:58:51.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673861 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T22:58:51.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.673873 I | ceph-object-zone-controller: successfully started 2022-03-08T22:58:51.665 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674020 I | ceph-object-controller: successfully started 2022-03-08T22:58:51.666 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674063 I | ceph-file-controller: successfully started 2022-03-08T22:58:51.666 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674093 I | ceph-nfs-controller: successfully started 2022-03-08T22:58:51.666 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674128 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T22:58:51.666 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674165 I | ceph-client-controller: successfully started 2022-03-08T22:58:51.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674190 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T22:58:51.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674224 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T22:58:51.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674243 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T22:58:51.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674258 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T22:58:51.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674273 I | ceph-bucket-topic: successfully started 2022-03-08T22:58:51.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674292 I | ceph-bucket-notification: successfully started 2022-03-08T22:58:51.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674313 I | ceph-bucket-notification: successfully started 2022-03-08T22:58:51.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.674325 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T22:58:51.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.675742 I | operator: starting the controller-runtime manager 2022-03-08T22:58:51.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.779996 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T22:58:51.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.780078 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T22:58:51.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.780100 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T22:58:51.773 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.781439 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:58:51.773 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.785446 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T22:58:51.775 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.787450 I | op-mon: parsing mon endpoints: a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 2022-03-08T22:58:51.776 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.787749 I | op-mon: parsing mon endpoints: a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 2022-03-08T22:58:51.776 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.787786 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T22:58:51.776 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.787796 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:58:51.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.789139 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T22:58:51.778 INFO:tasks.rook.operator.smithi042.stdout:I0308 22:58:51.789219 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:58:51.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.804390 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T22:58:51.794 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.807336 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:58:51.820 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833039 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T22:58:51.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833078 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T22:58:51.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833092 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T22:58:51.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833106 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T22:58:51.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833119 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T22:58:51.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833133 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T22:58:51.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833143 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T22:58:51.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833152 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T22:58:51.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833160 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T22:58:51.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833169 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T22:58:51.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833176 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T22:58:51.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833189 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T22:58:51.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833206 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T22:58:51.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833213 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T22:58:51.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833223 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T22:58:51.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833231 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T22:58:51.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833330 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:58:51.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:51.833355 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:58:51.985 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:58:52.009 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:58:52.009 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:58:57.762 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773453 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T22:58:57.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773493 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T22:58:57.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773504 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:58:57.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773513 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:58:57.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773520 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:58:57.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773528 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:58:57.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773535 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:58:57.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773541 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:58:57.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773550 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T22:58:57.765 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773561 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T22:58:57.765 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773572 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:58:57.765 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773582 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:58:57.765 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773593 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:58:57.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773604 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:58:57.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773614 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T22:58:57.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773628 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T22:58:57.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773642 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T22:58:57.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773653 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T22:58:57.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773663 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T22:58:57.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773672 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:58:57.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773685 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:58:57.767 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773697 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T22:58:57.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773709 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T22:58:57.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773726 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T22:58:57.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.773741 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T22:58:57.834 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.846919 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T22:58:57.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852612 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:58:57.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852629 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:58:57.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852634 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:58:57.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852638 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:58:57.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852642 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:58:57.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852647 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:58:57.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:57.852651 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T22:58:58.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.020949 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:58:58.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.020979 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:58:58.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.020987 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:58:58.062 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.075819 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T22:58:58.391 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.404620 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:58:58.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.404651 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:58:58.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.404660 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T22:58:58.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.507308 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:58:58.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.507336 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:58:58.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.507343 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:58:58.618 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.624598 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T22:58:58.781 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.793991 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:58:58.785 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.799054 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T22:58:58.786 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.799069 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:58:58.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:58.803492 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T22:58:59.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:59.932995 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:58:59.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:59.933023 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:58:59.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:59.933037 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:58:59.929 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:58:59.943091 I | op-mon: parsing mon endpoints: a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 2022-03-08T22:59:00.074 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.087389 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:59:00.076 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.087574 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:59:00.592 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.604934 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:59:00.682 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.694999 I | op-mon: start running mons 2022-03-08T22:59:00.706 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.719265 I | op-mon: parsing mon endpoints: a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 2022-03-08T22:59:00.732 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:00.744647 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789","10.110.240.83:6789","10.111.142.38:6789"]}] data:a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T22:59:01.068 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:01.081366 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:59:01.068 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:01.081554 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:59:02.011 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:02.572 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:02.592 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:59:02.593 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:02.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:02.683061 I | op-mon: targeting the mon count 3 2022-03-08T22:59:02.720 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:02.733187 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:59:03.094 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.107642 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:59:03.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.107690 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:59:03.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.442786 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:59:03.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.442820 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:59:03.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.791335 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:59:03.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:03.791370 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:59:04.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.181098 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:59:04.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.181126 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:59:04.509 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.522523 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:59:04.510 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.522542 I | op-mon: checking for basic quorum with existing mons 2022-03-08T22:59:04.603 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.616961 I | op-mon: mon "a" endpoint is [v2:10.100.208.21:3300,v1:10.100.208.21:6789] 2022-03-08T22:59:04.730 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:04.743467 I | op-mon: mon "b" endpoint is [v2:10.110.240.83:3300,v1:10.110.240.83:6789] 2022-03-08T22:59:05.072 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:05.085745 I | op-mon: mon "c" endpoint is [v2:10.111.142.38:3300,v1:10.111.142.38:6789] 2022-03-08T22:59:05.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:05.684957 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.208.21:6789","10.110.240.83:6789","10.111.142.38:6789"]}] data:c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T22:59:06.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:06.289063 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:59:06.280 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:06.289337 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:59:06.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:06.727389 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:59:06.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:06.737768 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:59:06.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:06.737796 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:59:07.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.288467 I | op-mon: mons running: [a b c] 2022-03-08T22:59:07.718 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.731402 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T22:59:07.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.739927 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T22:59:07.735 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.748868 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T22:59:07.735 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.748887 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:59:07.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:07.891213 I | op-mon: mons running: [a b c] 2022-03-08T22:59:08.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.301549 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T22:59:08.313 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.327007 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T22:59:08.323 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.336977 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T22:59:08.323 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.337000 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:59:08.474 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.487386 I | op-mon: mons running: [a b c] 2022-03-08T22:59:08.916 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.929205 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T22:59:08.916 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:08.929239 I | op-mon: mons created: 3 2022-03-08T22:59:09.307 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:09.320247 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T22:59:09.335 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:09.348274 I | op-mon: mons running: [a b c] 2022-03-08T22:59:09.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:09.762114 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T22:59:09.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:09.762161 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:59:10.165 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:10.178759 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:59:10.575 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:10.588564 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:59:11.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:11.026690 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:59:11.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:11.460881 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:59:11.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:11.460906 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:59:11.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.010813 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:59:12.404 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.417946 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:59:12.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.417976 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:59:12.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.417982 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:59:12.594 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:12.819 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.832817 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:59:12.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.865425 I | op-mgr: start running mgr 2022-03-08T22:59:12.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:12.865467 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:59:13.118 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:13.136 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:59:13.136 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:13.391 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.404951 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T22:59:13.402 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.415527 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T22:59:13.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.418987 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:59:13.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.546572 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T22:59:13.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.546709 I | op-mgr: successful modules: balancer 2022-03-08T22:59:13.558 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.566983 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:59:13.558 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.567014 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:59:13.574 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.588156 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:59:13.619 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.632417 I | op-osd: creating 1 new PVCs for device set "smithi042-lv-1" 2022-03-08T22:59:13.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.836449 I | op-osd: successfully provisioned PVC "smithi042-lv-1-data-09lqvf" 2022-03-08T22:59:13.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:13.836477 I | op-osd: creating 1 new PVCs for device set "smithi042-lv-2" 2022-03-08T22:59:14.033 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.046722 I | op-osd: successfully provisioned PVC "smithi042-lv-2-data-0khnqg" 2022-03-08T22:59:14.033 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.046756 I | op-osd: creating 1 new PVCs for device set "smithi042-lv-3" 2022-03-08T22:59:14.221 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.234594 I | op-osd: successfully provisioned PVC "smithi042-lv-3-data-084p6f" 2022-03-08T22:59:14.222 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.234633 I | op-osd: creating 1 new PVCs for device set "smithi042-lv-4" 2022-03-08T22:59:14.332 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.345231 I | op-mgr: successful modules: dashboard 2022-03-08T22:59:14.336 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.350324 I | op-mgr: successful modules: prometheus 2022-03-08T22:59:14.340 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.351073 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:59:14.421 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.434903 I | op-osd: successfully provisioned PVC "smithi042-lv-4-data-0rz4pz" 2022-03-08T22:59:14.422 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.434937 I | op-osd: creating 1 new PVCs for device set "smithi153-lv-1" 2022-03-08T22:59:14.621 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.634581 I | op-osd: successfully provisioned PVC "smithi153-lv-1-data-0vtz4j" 2022-03-08T22:59:14.622 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.634615 I | op-osd: creating 1 new PVCs for device set "smithi153-lv-2" 2022-03-08T22:59:14.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.834096 I | op-osd: successfully provisioned PVC "smithi153-lv-2-data-0qw442" 2022-03-08T22:59:14.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:14.834136 I | op-osd: creating 1 new PVCs for device set "smithi153-lv-3" 2022-03-08T22:59:15.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.033977 I | op-osd: successfully provisioned PVC "smithi153-lv-3-data-0bj679" 2022-03-08T22:59:15.021 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.034005 I | op-osd: creating 1 new PVCs for device set "smithi153-lv-4" 2022-03-08T22:59:15.222 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.235568 I | op-osd: successfully provisioned PVC "smithi153-lv-4-data-0h87v5" 2022-03-08T22:59:15.222 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.235606 I | op-osd: creating 1 new PVCs for device set "smithi178-lv-1" 2022-03-08T22:59:15.441 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.455105 I | op-osd: successfully provisioned PVC "smithi178-lv-1-data-0b2hpx" 2022-03-08T22:59:15.442 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.455150 I | op-osd: creating 1 new PVCs for device set "smithi178-lv-2" 2022-03-08T22:59:15.620 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.633636 I | op-osd: successfully provisioned PVC "smithi178-lv-2-data-0nv8kw" 2022-03-08T22:59:15.621 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.633677 I | op-osd: creating 1 new PVCs for device set "smithi178-lv-3" 2022-03-08T22:59:15.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.834721 I | op-osd: successfully provisioned PVC "smithi178-lv-3-data-0sf4l2" 2022-03-08T22:59:15.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.834765 I | op-osd: creating 1 new PVCs for device set "smithi178-lv-4" 2022-03-08T22:59:15.926 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:15.939097 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:59:16.021 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.034934 I | op-osd: successfully provisioned PVC "smithi178-lv-4-data-0scdxj" 2022-03-08T22:59:16.034 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.048520 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-1-data-09lqvf" 2022-03-08T22:59:16.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.497669 I | op-osd: started OSD provisioning job for PVC "smithi042-lv-1-data-09lqvf" 2022-03-08T22:59:16.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.497708 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-2-data-0khnqg" 2022-03-08T22:59:16.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.852175 I | op-osd: started OSD provisioning job for PVC "smithi042-lv-2-data-0khnqg" 2022-03-08T22:59:16.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:16.852212 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-3-data-084p6f" 2022-03-08T22:59:17.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:17.253332 I | op-osd: started OSD provisioning job for PVC "smithi042-lv-3-data-084p6f" 2022-03-08T22:59:17.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:17.253364 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-4-data-0rz4pz" 2022-03-08T22:59:17.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:17.692974 I | op-osd: started OSD provisioning job for PVC "smithi042-lv-4-data-0rz4pz" 2022-03-08T22:59:17.680 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:17.693007 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-1-data-0vtz4j" 2022-03-08T22:59:18.056 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:18.069917 I | op-osd: started OSD provisioning job for PVC "smithi153-lv-1-data-0vtz4j" 2022-03-08T22:59:18.057 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:18.069951 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-2-data-0qw442" 2022-03-08T22:59:19.032 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.042335 I | op-osd: started OSD provisioning job for PVC "smithi153-lv-2-data-0qw442" 2022-03-08T22:59:19.032 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.042382 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-3-data-0bj679" 2022-03-08T22:59:19.654 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.667249 I | op-osd: started OSD provisioning job for PVC "smithi153-lv-3-data-0bj679" 2022-03-08T22:59:19.654 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.667286 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-4-data-0h87v5" 2022-03-08T22:59:19.953 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.966620 I | op-osd: started OSD provisioning job for PVC "smithi153-lv-4-data-0h87v5" 2022-03-08T22:59:19.953 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:19.966654 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-1-data-0b2hpx" 2022-03-08T22:59:20.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.169968 I | op-osd: started OSD provisioning job for PVC "smithi178-lv-1-data-0b2hpx" 2022-03-08T22:59:20.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.170007 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-2-data-0nv8kw" 2022-03-08T22:59:20.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.340095 I | op-osd: started OSD provisioning job for PVC "smithi178-lv-2-data-0nv8kw" 2022-03-08T22:59:20.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.340129 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-3-data-0sf4l2" 2022-03-08T22:59:20.461 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.475068 I | op-osd: started OSD provisioning job for PVC "smithi178-lv-3-data-0sf4l2" 2022-03-08T22:59:20.462 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.475107 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-4-data-0scdxj" 2022-03-08T22:59:20.835 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.848788 I | op-osd: started OSD provisioning job for PVC "smithi178-lv-4-data-0scdxj" 2022-03-08T22:59:20.836 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.848826 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:59:20.836 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:20.848834 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:59:21.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.168916 I | op-osd: OSD orchestration status for PVC smithi042-lv-1-data-09lqvf is "starting" 2022-03-08T22:59:21.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.168957 I | op-osd: OSD orchestration status for PVC smithi042-lv-2-data-0khnqg is "starting" 2022-03-08T22:59:21.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.168975 I | op-osd: OSD orchestration status for PVC smithi042-lv-3-data-084p6f is "starting" 2022-03-08T22:59:21.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.168986 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0rz4pz is "starting" 2022-03-08T22:59:21.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.168997 I | op-osd: OSD orchestration status for PVC smithi153-lv-1-data-0vtz4j is "starting" 2022-03-08T22:59:21.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169008 I | op-osd: OSD orchestration status for PVC smithi153-lv-2-data-0qw442 is "starting" 2022-03-08T22:59:21.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169023 I | op-osd: OSD orchestration status for PVC smithi153-lv-3-data-0bj679 is "starting" 2022-03-08T22:59:21.158 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169038 I | op-osd: OSD orchestration status for PVC smithi153-lv-4-data-0h87v5 is "starting" 2022-03-08T22:59:21.158 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169056 I | op-osd: OSD orchestration status for PVC smithi178-lv-1-data-0b2hpx is "starting" 2022-03-08T22:59:21.159 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169083 I | op-osd: OSD orchestration status for PVC smithi178-lv-2-data-0nv8kw is "starting" 2022-03-08T22:59:21.159 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169104 I | op-osd: OSD orchestration status for PVC smithi178-lv-3-data-0sf4l2 is "starting" 2022-03-08T22:59:21.159 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:21.169129 I | op-osd: OSD orchestration status for PVC smithi178-lv-4-data-0scdxj is "starting" 2022-03-08T22:59:23.138 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:23.679 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:23.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:59:23.704 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:27.706 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:27.719361 I | op-osd: OSD orchestration status for node smithi042-lv-4-data-0rz4pz is "orchestrating" 2022-03-08T22:59:28.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:28.110761 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0rz4pz is "orchestrating" 2022-03-08T22:59:31.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:31.725204 I | op-osd: OSD orchestration status for node smithi153-lv-1-data-0vtz4j is "orchestrating" 2022-03-08T22:59:31.919 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:31.933185 I | op-osd: OSD orchestration status for node smithi153-lv-2-data-0qw442 is "orchestrating" 2022-03-08T22:59:32.465 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:32.478648 I | op-osd: OSD orchestration status for PVC smithi153-lv-1-data-0vtz4j is "orchestrating" 2022-03-08T22:59:32.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:32.709537 I | op-osd: OSD orchestration status for PVC smithi153-lv-2-data-0qw442 is "orchestrating" 2022-03-08T22:59:33.705 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:34.313 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:34.335 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":3,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1646780369,"num_remapped_pgs":0} 2022-03-08T22:59:34.336 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:35.285 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:35.298789 I | op-osd: OSD orchestration status for node smithi178-lv-2-data-0nv8kw is "orchestrating" 2022-03-08T22:59:35.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:35.299034 I | op-osd: OSD orchestration status for node smithi178-lv-3-data-0sf4l2 is "orchestrating" 2022-03-08T22:59:36.068 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:36.082485 I | op-osd: OSD orchestration status for PVC smithi178-lv-2-data-0nv8kw is "orchestrating" 2022-03-08T22:59:36.358 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:36.371138 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0rz4pz is "completed" 2022-03-08T22:59:36.358 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:36.371184 I | op-osd: creating OSD 0 on PVC "smithi042-lv-4-data-0rz4pz" 2022-03-08T22:59:36.358 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:36.371209 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-4-data-0rz4pz" 2022-03-08T22:59:37.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:37.247927 I | op-osd: OSD orchestration status for PVC smithi178-lv-3-data-0sf4l2 is "orchestrating" 2022-03-08T22:59:40.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:40.095029 I | op-osd: OSD orchestration status for node smithi178-lv-1-data-0b2hpx is "orchestrating" 2022-03-08T22:59:40.532 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:40.543074 I | op-osd: OSD orchestration status for PVC smithi153-lv-1-data-0vtz4j is "completed" 2022-03-08T22:59:40.532 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:40.543108 I | op-osd: creating OSD 1 on PVC "smithi153-lv-1-data-0vtz4j" 2022-03-08T22:59:40.532 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:40.543120 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-1-data-0vtz4j" 2022-03-08T22:59:42.865 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:42.878934 I | op-osd: OSD orchestration status for PVC smithi178-lv-1-data-0b2hpx is "orchestrating" 2022-03-08T22:59:42.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:42.879295 I | op-osd: OSD orchestration status for PVC smithi153-lv-2-data-0qw442 is "completed" 2022-03-08T22:59:42.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:42.879334 I | op-osd: creating OSD 2 on PVC "smithi153-lv-2-data-0qw442" 2022-03-08T22:59:42.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:42.879357 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-2-data-0qw442" 2022-03-08T22:59:44.338 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:44.933 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:44.947367 I | op-osd: OSD orchestration status for node smithi178-lv-4-data-0scdxj is "orchestrating" 2022-03-08T22:59:44.936 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:44.947847 I | op-osd: OSD orchestration status for PVC smithi178-lv-4-data-0scdxj is "orchestrating" 2022-03-08T22:59:44.937 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:44.948135 I | op-osd: OSD orchestration status for node smithi153-lv-3-data-0bj679 is "orchestrating" 2022-03-08T22:59:45.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:45.253312 I | op-osd: OSD orchestration status for PVC smithi153-lv-3-data-0bj679 is "orchestrating" 2022-03-08T22:59:45.332 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:45.346571 I | op-osd: OSD orchestration status for node smithi153-lv-4-data-0h87v5 is "orchestrating" 2022-03-08T22:59:46.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:46.280 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":9,"num_osds":7,"num_up_osds":0,"osd_up_since":0,"num_in_osds":7,"osd_in_since":1646780385,"num_remapped_pgs":0} 2022-03-08T22:59:46.280 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:46.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:46.838794 I | op-osd: OSD orchestration status for PVC smithi153-lv-4-data-0h87v5 is "orchestrating" 2022-03-08T22:59:47.027 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.040811 I | op-osd: OSD orchestration status for PVC smithi178-lv-3-data-0sf4l2 is "completed" 2022-03-08T22:59:47.027 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.040839 I | op-osd: creating OSD 3 on PVC "smithi178-lv-3-data-0sf4l2" 2022-03-08T22:59:47.027 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.040852 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-3-data-0sf4l2" 2022-03-08T22:59:47.689 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.703224 I | op-osd: OSD orchestration status for PVC smithi178-lv-2-data-0nv8kw is "completed" 2022-03-08T22:59:47.689 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.703246 I | op-osd: creating OSD 4 on PVC "smithi178-lv-2-data-0nv8kw" 2022-03-08T22:59:47.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.703258 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-2-data-0nv8kw" 2022-03-08T22:59:47.979 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:47.993354 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi178": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:59:48.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:48.806346 I | op-osd: OSD orchestration status for node smithi042-lv-3-data-084p6f is "orchestrating" 2022-03-08T22:59:48.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:48.806413 I | op-osd: OSD orchestration status for node smithi042-lv-1-data-09lqvf is "orchestrating" 2022-03-08T22:59:49.470 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:49.484290 I | op-osd: OSD orchestration status for PVC smithi178-lv-1-data-0b2hpx is "completed" 2022-03-08T22:59:49.471 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:49.484323 I | op-osd: creating OSD 5 on PVC "smithi178-lv-1-data-0b2hpx" 2022-03-08T22:59:49.471 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:49.484346 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-1-data-0b2hpx" 2022-03-08T22:59:49.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:50.002312 I | op-osd: OSD orchestration status for PVC smithi042-lv-1-data-09lqvf is "orchestrating" 2022-03-08T22:59:49.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:50.002644 I | op-osd: OSD orchestration status for PVC smithi042-lv-3-data-084p6f is "orchestrating" 2022-03-08T22:59:49.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:50.008857 I | op-osd: OSD orchestration status for node smithi042-lv-2-data-0khnqg is "orchestrating" 2022-03-08T22:59:50.954 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:50.968510 I | op-osd: OSD orchestration status for PVC smithi042-lv-2-data-0khnqg is "orchestrating" 2022-03-08T22:59:51.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:51.844393 I | op-osd: OSD orchestration status for PVC smithi178-lv-4-data-0scdxj is "completed" 2022-03-08T22:59:51.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:51.844418 I | op-osd: creating OSD 6 on PVC "smithi178-lv-4-data-0scdxj" 2022-03-08T22:59:51.831 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:51.844433 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-4-data-0scdxj" 2022-03-08T22:59:53.528 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:53.542844 I | op-osd: OSD orchestration status for PVC smithi153-lv-3-data-0bj679 is "completed" 2022-03-08T22:59:53.529 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:53.542866 I | op-osd: creating OSD 7 on PVC "smithi153-lv-3-data-0bj679" 2022-03-08T22:59:53.529 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:53.542875 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-3-data-0bj679" 2022-03-08T22:59:55.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:55.161545 I | op-osd: OSD orchestration status for PVC smithi153-lv-4-data-0h87v5 is "completed" 2022-03-08T22:59:55.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:55.161568 I | op-osd: creating OSD 8 on PVC "smithi153-lv-4-data-0h87v5" 2022-03-08T22:59:55.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:55.161577 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-4-data-0h87v5" 2022-03-08T22:59:56.282 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T22:59:57.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T22:59:57.932 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646780394,"num_remapped_pgs":0} 2022-03-08T22:59:57.933 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T22:59:58.459 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:58.473802 I | op-osd: OSD orchestration status for PVC smithi042-lv-1-data-09lqvf is "completed" 2022-03-08T22:59:58.460 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:58.473842 I | op-osd: creating OSD 9 on PVC "smithi042-lv-1-data-09lqvf" 2022-03-08T22:59:58.460 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:58.473856 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-1-data-09lqvf" 2022-03-08T22:59:59.682 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:59.696896 I | op-osd: OSD orchestration status for PVC smithi042-lv-3-data-084p6f is "completed" 2022-03-08T22:59:59.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:59.696929 I | op-osd: creating OSD 10 on PVC "smithi042-lv-3-data-084p6f" 2022-03-08T22:59:59.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 22:59:59.696943 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-3-data-084p6f" 2022-03-08T23:00:01.919 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:00:01.931259 I | op-osd: OSD orchestration status for PVC smithi042-lv-2-data-0khnqg is "completed" 2022-03-08T23:00:01.919 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:00:01.931292 I | op-osd: creating OSD 11 on PVC "smithi042-lv-2-data-0khnqg" 2022-03-08T23:00:01.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:00:01.931305 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-2-data-0khnqg" 2022-03-08T23:00:04.471 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:00:04.485790 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:00:04.472 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:00:04.485810 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:07.934 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T23:00:09.805 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:09.827 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":17,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646780394,"num_remapped_pgs":0} 2022-03-08T23:00:09.828 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:00:19.830 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T23:00:20.596 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:20.616 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":22,"num_osds":12,"num_up_osds":2,"osd_up_since":1646780411,"num_in_osds":12,"osd_in_since":1646780394,"num_remapped_pgs":0} 2022-03-08T23:00:20.617 INFO:tasks.rook: have 2/12 OSDs 2022-03-08T23:00:30.618 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T23:00:31.126 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:31.149 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":30,"num_osds":12,"num_up_osds":9,"osd_up_since":1646780426,"num_in_osds":12,"osd_in_since":1646780394,"num_remapped_pgs":0} 2022-03-08T23:00:31.150 INFO:tasks.rook: have 9/12 OSDs 2022-03-08T23:00:41.151 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd stat -f json 2022-03-08T23:00:41.789 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:41.812 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"num_osds":12,"num_up_osds":12,"osd_up_since":1646780436,"num_in_osds":12,"osd_in_since":1646780394,"num_remapped_pgs":0} 2022-03-08T23:00:41.813 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T23:00:41.813 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- cat /etc/ceph/ceph.conf 2022-03-08T23:00:42.091 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2022-03-08T23:00:42.092 INFO:teuthology.orchestra.run.smithi042.stdout:mon_host = 10.111.142.38:6789,10.100.208.21:6789,10.110.240.83:6789 2022-03-08T23:00:42.092 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:42.092 INFO:teuthology.orchestra.run.smithi042.stdout:[client.admin] 2022-03-08T23:00:42.093 INFO:teuthology.orchestra.run.smithi042.stdout:keyring = /etc/ceph/keyring 2022-03-08T23:00:42.096 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- cat /etc/ceph/keyring 2022-03-08T23:00:42.290 INFO:teuthology.orchestra.run.smithi042.stdout:[client.admin] 2022-03-08T23:00:42.291 INFO:teuthology.orchestra.run.smithi042.stdout:key = AQDh3idiUHdDBBAAumDqUjmLlKeO289iTVnZxA== 2022-03-08T23:00:42.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph 2022-03-08T23:00:42.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2022-03-08T23:00:42.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2022-03-08T23:00:42.343 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T23:00:42.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:00:42.359 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T23:00:42.359 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:00:42.372 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T23:00:42.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:00:42.389 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T23:00:42.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:00:42.407 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T23:00:42.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:00:42.424 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T23:00:42.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:00:42.487 INFO:tasks.rook:Setting up client nodes... 2022-03-08T23:00:42.488 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:00:43.290 INFO:teuthology.orchestra.run.smithi042.stdout:[client.a] 2022-03-08T23:00:43.291 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAb4CdiOA8bCxAADyP1YgJ26IXAs9NwosEqbw== 2022-03-08T23:00:43.313 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2022-03-08T23:00:43.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T23:00:43.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T23:00:43.338 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:00:44.162 INFO:teuthology.orchestra.run.smithi042.stdout:[client.b] 2022-03-08T23:00:44.163 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAb4Cdi2umBOBAA3zVVW6qg0+hX9/oHcQ+57A== 2022-03-08T23:00:44.195 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2022-03-08T23:00:44.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-08T23:00:44.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-08T23:00:44.218 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:00:44.917 INFO:teuthology.orchestra.run.smithi042.stdout:[client.c] 2022-03-08T23:00:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAc4CdiRnQuMBAAcJ8VQFOYksgPRX5bkEXoUw== 2022-03-08T23:00:44.939 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2022-03-08T23:00:44.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-08T23:00:44.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-08T23:00:44.962 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd dump --format=json 2022-03-08T23:00:45.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:45.505 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"fsid":"20eb5159-27d7-4658-84d5-b16f4f92328a","created":"2022-03-08T22:55:42.884079+0000","modified":"2022-03-08T23:00:37.935324+0000","last_up_change":"2022-03-08T23:00:36.690679+0000","last_in_change":"2022-03-08T22:59:54.545158+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:00:22.658772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"38513cb7-6bae-42c2-96a0-45f590df54e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6800","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6801","nonce":2388217260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6802","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6803","nonce":2388217260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6806","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6807","nonce":2388217260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6804","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6805","nonce":2388217260}]},"public_addr":"10.249.72.10:6801/2388217260","cluster_addr":"10.249.72.10:6803/2388217260","heartbeat_back_addr":"10.249.72.10:6807/2388217260","heartbeat_front_addr":"10.249.72.10:6805/2388217260","state":["exists","up"]},{"osd":1,"uuid":"2141c595-12a7-43fa-b63d-5c544ca1fbdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6800","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6801","nonce":2097104474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6802","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6803","nonce":2097104474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6806","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6807","nonce":2097104474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6804","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6805","nonce":2097104474}]},"public_addr":"10.249.74.13:6801/2097104474","cluster_addr":"10.249.74.13:6803/2097104474","heartbeat_back_addr":"10.249.74.13:6807/2097104474","heartbeat_front_addr":"10.249.74.13:6805/2097104474","state":["exists","up"]},{"osd":2,"uuid":"9c7208af-8051-4857-b99f-34a29c108b35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6800","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6801","nonce":4046926506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6802","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6803","nonce":4046926506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6806","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6807","nonce":4046926506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6804","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6805","nonce":4046926506}]},"public_addr":"10.249.74.14:6801/4046926506","cluster_addr":"10.249.74.14:6803/4046926506","heartbeat_back_addr":"10.249.74.14:6807/4046926506","heartbeat_front_addr":"10.249.74.14:6805/4046926506","state":["exists","up"]},{"osd":3,"uuid":"3ceb23c4-0fbf-4c62-912b-4ea049d24a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6800","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6801","nonce":1155508313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6802","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6803","nonce":1155508313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6806","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6807","nonce":1155508313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6804","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6805","nonce":1155508313}]},"public_addr":"10.249.73.18:6801/1155508313","cluster_addr":"10.249.73.18:6803/1155508313","heartbeat_back_addr":"10.249.73.18:6807/1155508313","heartbeat_front_addr":"10.249.73.18:6805/1155508313","state":["exists","up"]},{"osd":4,"uuid":"143786de-8b1e-4699-8337-3c9394b023eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6800","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6801","nonce":2697649584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6802","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6803","nonce":2697649584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6806","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6807","nonce":2697649584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6804","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6805","nonce":2697649584}]},"public_addr":"10.249.73.19:6801/2697649584","cluster_addr":"10.249.73.19:6803/2697649584","heartbeat_back_addr":"10.249.73.19:6807/2697649584","heartbeat_front_addr":"10.249.73.19:6805/2697649584","state":["exists","up"]},{"osd":5,"uuid":"55fc3ab0-474b-4d84-99c1-6527affdae27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6800","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6801","nonce":1439296624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6802","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6803","nonce":1439296624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6806","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6807","nonce":1439296624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6804","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6805","nonce":1439296624}]},"public_addr":"10.249.73.20:6801/1439296624","cluster_addr":"10.249.73.20:6803/1439296624","heartbeat_back_addr":"10.249.73.20:6807/1439296624","heartbeat_front_addr":"10.249.73.20:6805/1439296624","state":["exists","up"]},{"osd":6,"uuid":"9b4034ea-ccf4-4aff-909d-d0c88ad6b112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6800","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6801","nonce":1013886359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6802","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6803","nonce":1013886359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6806","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6807","nonce":1013886359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6804","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6805","nonce":1013886359}]},"public_addr":"10.249.73.21:6801/1013886359","cluster_addr":"10.249.73.21:6803/1013886359","heartbeat_back_addr":"10.249.73.21:6807/1013886359","heartbeat_front_addr":"10.249.73.21:6805/1013886359","state":["exists","up"]},{"osd":7,"uuid":"bb7fd008-9682-4e4f-ba5a-c816eeca7193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6800","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6801","nonce":2175473542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6802","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6803","nonce":2175473542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6806","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6807","nonce":2175473542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6804","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6805","nonce":2175473542}]},"public_addr":"10.249.74.15:6801/2175473542","cluster_addr":"10.249.74.15:6803/2175473542","heartbeat_back_addr":"10.249.74.15:6807/2175473542","heartbeat_front_addr":"10.249.74.15:6805/2175473542","state":["exists","up"]},{"osd":8,"uuid":"e76c0b63-576b-4837-ac4b-9c7cb3bd9b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6800","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6801","nonce":2915904220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6802","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6803","nonce":2915904220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6806","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6807","nonce":2915904220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6804","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6805","nonce":2915904220}]},"public_addr":"10.249.74.16:6801/2915904220","cluster_addr":"10.249.74.16:6803/2915904220","heartbeat_back_addr":"10.249.74.16:6807/2915904220","heartbeat_front_addr":"10.249.74.16:6805/2915904220","state":["exists","up"]},{"osd":9,"uuid":"b8dfb7e8-db1f-4b52-a706-97e444ccd9de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6800","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6801","nonce":490728162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6802","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6803","nonce":490728162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6806","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6807","nonce":490728162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6804","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6805","nonce":490728162}]},"public_addr":"10.249.72.11:6801/490728162","cluster_addr":"10.249.72.11:6803/490728162","heartbeat_back_addr":"10.249.72.11:6807/490728162","heartbeat_front_addr":"10.249.72.11:6805/490728162","state":["exists","up"]},{"osd":10,"uuid":"8c880fe6-414e-42a6-a120-f18d35fc5479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6800","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6801","nonce":3346958620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6802","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6803","nonce":3346958620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6806","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6807","nonce":3346958620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6804","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6805","nonce":3346958620}]},"public_addr":"10.249.72.12:6801/3346958620","cluster_addr":"10.249.72.12:6803/3346958620","heartbeat_back_addr":"10.249.72.12:6807/3346958620","heartbeat_front_addr":"10.249.72.12:6805/3346958620","state":["exists","up"]},{"osd":11,"uuid":"adc5fa12-62b8-4205-9cb7-b0a5a5290dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6800","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6801","nonce":3244507661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6802","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6803","nonce":3244507661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6806","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6807","nonce":3244507661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6804","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6805","nonce":3244507661}]},"public_addr":"10.249.72.13:6801/3244507661","cluster_addr":"10.249.72.13:6803/3244507661","heartbeat_back_addr":"10.249.72.13:6807/3244507661","heartbeat_front_addr":"10.249.72.13:6805/3244507661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:03.943852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:09.486679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:14.045283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.116250+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.957966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.172344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.101473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:21.496558+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.291139+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:30.213093+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.421871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.363178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.73.10:6801/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:6800/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1449118396":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/560471415":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1955487323":"2022-03-09T22:58:36.357970+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:00:45.519 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T23:00:22.658772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T23:00:45.520 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd pool get .mgr pg_num 2022-03-08T23:00:46.006 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 1 2022-03-08T23:00:46.025 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T23:00:46.026 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T23:00:46.026 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph mgr dump --format=json 2022-03-08T23:00:46.583 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:46.601 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":12,"active_gid":14290,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.10:6800","nonce":3363657004},{"type":"v1","addr":"10.249.73.10:6801","nonce":3363657004}]},"active_addr":"10.249.73.10:6801/3363657004","active_change":"2022-03-08T22:58:36.396466+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.249.73.10:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.249.73.10:0","nonce":1918381579}]},{"addrvec":[{"type":"v2","addr":"10.249.73.10:0","nonce":588563200}]},{"addrvec":[{"type":"v2","addr":"10.249.73.10:0","nonce":2544908242}]},{"addrvec":[{"type":"v2","addr":"10.249.73.10:0","nonce":3292006576}]}]}} 2022-03-08T23:00:46.604 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T23:00:46.604 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T23:00:46.604 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd dump --format=json 2022-03-08T23:00:47.120 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:47.121 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"fsid":"20eb5159-27d7-4658-84d5-b16f4f92328a","created":"2022-03-08T22:55:42.884079+0000","modified":"2022-03-08T23:00:37.935324+0000","last_up_change":"2022-03-08T23:00:36.690679+0000","last_in_change":"2022-03-08T22:59:54.545158+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:00:22.658772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"38513cb7-6bae-42c2-96a0-45f590df54e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6800","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6801","nonce":2388217260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6802","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6803","nonce":2388217260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6806","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6807","nonce":2388217260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6804","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6805","nonce":2388217260}]},"public_addr":"10.249.72.10:6801/2388217260","cluster_addr":"10.249.72.10:6803/2388217260","heartbeat_back_addr":"10.249.72.10:6807/2388217260","heartbeat_front_addr":"10.249.72.10:6805/2388217260","state":["exists","up"]},{"osd":1,"uuid":"2141c595-12a7-43fa-b63d-5c544ca1fbdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6800","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6801","nonce":2097104474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6802","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6803","nonce":2097104474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6806","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6807","nonce":2097104474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6804","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6805","nonce":2097104474}]},"public_addr":"10.249.74.13:6801/2097104474","cluster_addr":"10.249.74.13:6803/2097104474","heartbeat_back_addr":"10.249.74.13:6807/2097104474","heartbeat_front_addr":"10.249.74.13:6805/2097104474","state":["exists","up"]},{"osd":2,"uuid":"9c7208af-8051-4857-b99f-34a29c108b35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6800","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6801","nonce":4046926506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6802","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6803","nonce":4046926506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6806","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6807","nonce":4046926506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6804","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6805","nonce":4046926506}]},"public_addr":"10.249.74.14:6801/4046926506","cluster_addr":"10.249.74.14:6803/4046926506","heartbeat_back_addr":"10.249.74.14:6807/4046926506","heartbeat_front_addr":"10.249.74.14:6805/4046926506","state":["exists","up"]},{"osd":3,"uuid":"3ceb23c4-0fbf-4c62-912b-4ea049d24a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6800","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6801","nonce":1155508313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6802","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6803","nonce":1155508313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6806","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6807","nonce":1155508313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6804","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6805","nonce":1155508313}]},"public_addr":"10.249.73.18:6801/1155508313","cluster_addr":"10.249.73.18:6803/1155508313","heartbeat_back_addr":"10.249.73.18:6807/1155508313","heartbeat_front_addr":"10.249.73.18:6805/1155508313","state":["exists","up"]},{"osd":4,"uuid":"143786de-8b1e-4699-8337-3c9394b023eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6800","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6801","nonce":2697649584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6802","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6803","nonce":2697649584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6806","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6807","nonce":2697649584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6804","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6805","nonce":2697649584}]},"public_addr":"10.249.73.19:6801/2697649584","cluster_addr":"10.249.73.19:6803/2697649584","heartbeat_back_addr":"10.249.73.19:6807/2697649584","heartbeat_front_addr":"10.249.73.19:6805/2697649584","state":["exists","up"]},{"osd":5,"uuid":"55fc3ab0-474b-4d84-99c1-6527affdae27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6800","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6801","nonce":1439296624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6802","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6803","nonce":1439296624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6806","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6807","nonce":1439296624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6804","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6805","nonce":1439296624}]},"public_addr":"10.249.73.20:6801/1439296624","cluster_addr":"10.249.73.20:6803/1439296624","heartbeat_back_addr":"10.249.73.20:6807/1439296624","heartbeat_front_addr":"10.249.73.20:6805/1439296624","state":["exists","up"]},{"osd":6,"uuid":"9b4034ea-ccf4-4aff-909d-d0c88ad6b112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6800","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6801","nonce":1013886359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6802","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6803","nonce":1013886359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6806","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6807","nonce":1013886359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6804","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6805","nonce":1013886359}]},"public_addr":"10.249.73.21:6801/1013886359","cluster_addr":"10.249.73.21:6803/1013886359","heartbeat_back_addr":"10.249.73.21:6807/1013886359","heartbeat_front_addr":"10.249.73.21:6805/1013886359","state":["exists","up"]},{"osd":7,"uuid":"bb7fd008-9682-4e4f-ba5a-c816eeca7193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6800","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6801","nonce":2175473542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6802","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6803","nonce":2175473542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6806","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6807","nonce":2175473542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6804","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6805","nonce":2175473542}]},"public_addr":"10.249.74.15:6801/2175473542","cluster_addr":"10.249.74.15:6803/2175473542","heartbeat_back_addr":"10.249.74.15:6807/2175473542","heartbeat_front_addr":"10.249.74.15:6805/2175473542","state":["exists","up"]},{"osd":8,"uuid":"e76c0b63-576b-4837-ac4b-9c7cb3bd9b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6800","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6801","nonce":2915904220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6802","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6803","nonce":2915904220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6806","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6807","nonce":2915904220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6804","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6805","nonce":2915904220}]},"public_addr":"10.249.74.16:6801/2915904220","cluster_addr":"10.249.74.16:6803/2915904220","heartbeat_back_addr":"10.249.74.16:6807/2915904220","heartbeat_front_addr":"10.249.74.16:6805/2915904220","state":["exists","up"]},{"osd":9,"uuid":"b8dfb7e8-db1f-4b52-a706-97e444ccd9de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6800","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6801","nonce":490728162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6802","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6803","nonce":490728162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6806","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6807","nonce":490728162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6804","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6805","nonce":490728162}]},"public_addr":"10.249.72.11:6801/490728162","cluster_addr":"10.249.72.11:6803/490728162","heartbeat_back_addr":"10.249.72.11:6807/490728162","heartbeat_front_addr":"10.249.72.11:6805/490728162","state":["exists","up"]},{"osd":10,"uuid":"8c880fe6-414e-42a6-a120-f18d35fc5479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6800","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6801","nonce":3346958620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6802","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6803","nonce":3346958620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6806","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6807","nonce":3346958620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6804","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6805","nonce":3346958620}]},"public_addr":"10.249.72.12:6801/3346958620","cluster_addr":"10.249.72.12:6803/3346958620","heartbeat_back_addr":"10.249.72.12:6807/3346958620","heartbeat_front_addr":"10.249.72.12:6805/3346958620","state":["exists","up"]},{"osd":11,"uuid":"adc5fa12-62b8-4205-9cb7-b0a5a5290dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6800","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6801","nonce":3244507661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6802","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6803","nonce":3244507661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6806","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6807","nonce":3244507661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6804","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6805","nonce":3244507661}]},"public_addr":"10.249.72.13:6801/3244507661","cluster_addr":"10.249.72.13:6803/3244507661","heartbeat_back_addr":"10.249.72.13:6807/3244507661","heartbeat_front_addr":"10.249.72.13:6805/3244507661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:03.943852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:09.486679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:14.045283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.116250+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.957966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.172344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.101473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:21.496558+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.291139+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:30.213093+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.421871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.363178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.73.10:6801/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:6800/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1449118396":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/560471415":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1955487323":"2022-03-09T22:58:36.357970+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:00:47.139 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T23:00:47.140 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd dump --format=json 2022-03-08T23:00:47.644 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:47.644 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"fsid":"20eb5159-27d7-4658-84d5-b16f4f92328a","created":"2022-03-08T22:55:42.884079+0000","modified":"2022-03-08T23:00:37.935324+0000","last_up_change":"2022-03-08T23:00:36.690679+0000","last_in_change":"2022-03-08T22:59:54.545158+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:00:22.658772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"38513cb7-6bae-42c2-96a0-45f590df54e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6800","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6801","nonce":2388217260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6802","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6803","nonce":2388217260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6806","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6807","nonce":2388217260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.10:6804","nonce":2388217260},{"type":"v1","addr":"10.249.72.10:6805","nonce":2388217260}]},"public_addr":"10.249.72.10:6801/2388217260","cluster_addr":"10.249.72.10:6803/2388217260","heartbeat_back_addr":"10.249.72.10:6807/2388217260","heartbeat_front_addr":"10.249.72.10:6805/2388217260","state":["exists","up"]},{"osd":1,"uuid":"2141c595-12a7-43fa-b63d-5c544ca1fbdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6800","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6801","nonce":2097104474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6802","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6803","nonce":2097104474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6806","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6807","nonce":2097104474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.13:6804","nonce":2097104474},{"type":"v1","addr":"10.249.74.13:6805","nonce":2097104474}]},"public_addr":"10.249.74.13:6801/2097104474","cluster_addr":"10.249.74.13:6803/2097104474","heartbeat_back_addr":"10.249.74.13:6807/2097104474","heartbeat_front_addr":"10.249.74.13:6805/2097104474","state":["exists","up"]},{"osd":2,"uuid":"9c7208af-8051-4857-b99f-34a29c108b35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6800","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6801","nonce":4046926506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6802","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6803","nonce":4046926506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6806","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6807","nonce":4046926506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.14:6804","nonce":4046926506},{"type":"v1","addr":"10.249.74.14:6805","nonce":4046926506}]},"public_addr":"10.249.74.14:6801/4046926506","cluster_addr":"10.249.74.14:6803/4046926506","heartbeat_back_addr":"10.249.74.14:6807/4046926506","heartbeat_front_addr":"10.249.74.14:6805/4046926506","state":["exists","up"]},{"osd":3,"uuid":"3ceb23c4-0fbf-4c62-912b-4ea049d24a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6800","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6801","nonce":1155508313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6802","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6803","nonce":1155508313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6806","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6807","nonce":1155508313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.18:6804","nonce":1155508313},{"type":"v1","addr":"10.249.73.18:6805","nonce":1155508313}]},"public_addr":"10.249.73.18:6801/1155508313","cluster_addr":"10.249.73.18:6803/1155508313","heartbeat_back_addr":"10.249.73.18:6807/1155508313","heartbeat_front_addr":"10.249.73.18:6805/1155508313","state":["exists","up"]},{"osd":4,"uuid":"143786de-8b1e-4699-8337-3c9394b023eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6800","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6801","nonce":2697649584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6802","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6803","nonce":2697649584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6806","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6807","nonce":2697649584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.19:6804","nonce":2697649584},{"type":"v1","addr":"10.249.73.19:6805","nonce":2697649584}]},"public_addr":"10.249.73.19:6801/2697649584","cluster_addr":"10.249.73.19:6803/2697649584","heartbeat_back_addr":"10.249.73.19:6807/2697649584","heartbeat_front_addr":"10.249.73.19:6805/2697649584","state":["exists","up"]},{"osd":5,"uuid":"55fc3ab0-474b-4d84-99c1-6527affdae27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6800","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6801","nonce":1439296624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6802","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6803","nonce":1439296624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6806","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6807","nonce":1439296624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.20:6804","nonce":1439296624},{"type":"v1","addr":"10.249.73.20:6805","nonce":1439296624}]},"public_addr":"10.249.73.20:6801/1439296624","cluster_addr":"10.249.73.20:6803/1439296624","heartbeat_back_addr":"10.249.73.20:6807/1439296624","heartbeat_front_addr":"10.249.73.20:6805/1439296624","state":["exists","up"]},{"osd":6,"uuid":"9b4034ea-ccf4-4aff-909d-d0c88ad6b112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6800","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6801","nonce":1013886359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6802","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6803","nonce":1013886359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6806","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6807","nonce":1013886359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.73.21:6804","nonce":1013886359},{"type":"v1","addr":"10.249.73.21:6805","nonce":1013886359}]},"public_addr":"10.249.73.21:6801/1013886359","cluster_addr":"10.249.73.21:6803/1013886359","heartbeat_back_addr":"10.249.73.21:6807/1013886359","heartbeat_front_addr":"10.249.73.21:6805/1013886359","state":["exists","up"]},{"osd":7,"uuid":"bb7fd008-9682-4e4f-ba5a-c816eeca7193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6800","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6801","nonce":2175473542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6802","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6803","nonce":2175473542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6806","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6807","nonce":2175473542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.15:6804","nonce":2175473542},{"type":"v1","addr":"10.249.74.15:6805","nonce":2175473542}]},"public_addr":"10.249.74.15:6801/2175473542","cluster_addr":"10.249.74.15:6803/2175473542","heartbeat_back_addr":"10.249.74.15:6807/2175473542","heartbeat_front_addr":"10.249.74.15:6805/2175473542","state":["exists","up"]},{"osd":8,"uuid":"e76c0b63-576b-4837-ac4b-9c7cb3bd9b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6800","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6801","nonce":2915904220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6802","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6803","nonce":2915904220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6806","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6807","nonce":2915904220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.74.16:6804","nonce":2915904220},{"type":"v1","addr":"10.249.74.16:6805","nonce":2915904220}]},"public_addr":"10.249.74.16:6801/2915904220","cluster_addr":"10.249.74.16:6803/2915904220","heartbeat_back_addr":"10.249.74.16:6807/2915904220","heartbeat_front_addr":"10.249.74.16:6805/2915904220","state":["exists","up"]},{"osd":9,"uuid":"b8dfb7e8-db1f-4b52-a706-97e444ccd9de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6800","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6801","nonce":490728162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6802","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6803","nonce":490728162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6806","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6807","nonce":490728162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.11:6804","nonce":490728162},{"type":"v1","addr":"10.249.72.11:6805","nonce":490728162}]},"public_addr":"10.249.72.11:6801/490728162","cluster_addr":"10.249.72.11:6803/490728162","heartbeat_back_addr":"10.249.72.11:6807/490728162","heartbeat_front_addr":"10.249.72.11:6805/490728162","state":["exists","up"]},{"osd":10,"uuid":"8c880fe6-414e-42a6-a120-f18d35fc5479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6800","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6801","nonce":3346958620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6802","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6803","nonce":3346958620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6806","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6807","nonce":3346958620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.12:6804","nonce":3346958620},{"type":"v1","addr":"10.249.72.12:6805","nonce":3346958620}]},"public_addr":"10.249.72.12:6801/3346958620","cluster_addr":"10.249.72.12:6803/3346958620","heartbeat_back_addr":"10.249.72.12:6807/3346958620","heartbeat_front_addr":"10.249.72.12:6805/3346958620","state":["exists","up"]},{"osd":11,"uuid":"adc5fa12-62b8-4205-9cb7-b0a5a5290dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6800","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6801","nonce":3244507661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6802","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6803","nonce":3244507661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6806","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6807","nonce":3244507661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.72.13:6804","nonce":3244507661},{"type":"v1","addr":"10.249.72.13:6805","nonce":3244507661}]},"public_addr":"10.249.72.13:6801/3244507661","cluster_addr":"10.249.72.13:6803/3244507661","heartbeat_back_addr":"10.249.72.13:6807/3244507661","heartbeat_front_addr":"10.249.72.13:6805/3244507661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:03.943852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:09.486679+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:14.045283+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.116250+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.957966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.172344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:23.101473+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:21.496558+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:22.291139+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:30.213093+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.421871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:00:34.363178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.73.10:6801/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:6800/4108680690":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1449118396":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/560471415":"2022-03-09T22:58:36.357970+0000","10.249.73.10:0/1955487323":"2022-03-09T22:58:36.357970+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:00:47.667 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.0 flush_pg_stats 2022-03-08T23:00:47.667 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.1 flush_pg_stats 2022-03-08T23:00:47.668 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.2 flush_pg_stats 2022-03-08T23:00:47.669 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.3 flush_pg_stats 2022-03-08T23:00:47.669 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.4 flush_pg_stats 2022-03-08T23:00:47.670 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.5 flush_pg_stats 2022-03-08T23:00:47.671 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.6 flush_pg_stats 2022-03-08T23:00:47.671 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.7 flush_pg_stats 2022-03-08T23:00:47.672 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.8 flush_pg_stats 2022-03-08T23:00:47.673 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.9 flush_pg_stats 2022-03-08T23:00:47.673 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.10 flush_pg_stats 2022-03-08T23:00:47.674 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph tell osd.11 flush_pg_stats 2022-03-08T23:00:48.202 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:48.203 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.5 2022-03-08T23:00:48.580 INFO:teuthology.orchestra.run.smithi042.stdout:81604378634 2022-03-08T23:00:48.581 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.1 2022-03-08T23:00:48.590 INFO:teuthology.orchestra.run.smithi042.stdout:77309411340 2022-03-08T23:00:48.591 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.0 2022-03-08T23:00:48.592 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:48.592 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.6 2022-03-08T23:00:48.593 INFO:teuthology.orchestra.run.smithi042.stdout:137438953476 2022-03-08T23:00:48.594 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.9 2022-03-08T23:00:48.595 INFO:teuthology.orchestra.run.smithi042.stdout:111669149703 2022-03-08T23:00:48.596 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.7 2022-03-08T23:00:48.598 INFO:teuthology.orchestra.run.smithi042.stdout:98784247817 2022-03-08T23:00:48.599 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.2 2022-03-08T23:00:48.600 INFO:teuthology.orchestra.run.smithi042.stdout:150323855365 2022-03-08T23:00:48.601 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.10 2022-03-08T23:00:48.604 INFO:teuthology.orchestra.run.smithi042.stdout:111669149702 2022-03-08T23:00:48.605 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.8 2022-03-08T23:00:48.640 INFO:teuthology.orchestra.run.smithi042.stdout:115964116999 2022-03-08T23:00:48.641 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.4 2022-03-08T23:00:48.669 INFO:teuthology.orchestra.run.smithi042.stdout:115964116999 2022-03-08T23:00:48.669 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.3 2022-03-08T23:00:48.797 INFO:teuthology.orchestra.run.smithi042.stdout:150323855364 2022-03-08T23:00:48.797 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.11 2022-03-08T23:00:49.164 INFO:teuthology.orchestra.run.smithi042.stdout:115964116997 2022-03-08T23:00:49.347 INFO:teuthology.orchestra.run.smithi042.stdout:81604378633 2022-03-08T23:00:49.352 INFO:teuthology.orchestra.run.smithi042.stdout:137438953475 2022-03-08T23:00:49.372 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116997 for osd.5 2022-03-08T23:00:49.430 INFO:teuthology.orchestra.run.smithi042.stdout:111669149701 2022-03-08T23:00:49.477 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378634 got 81604378633 for osd.1 2022-03-08T23:00:49.482 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953476 got 137438953475 for osd.9 2022-03-08T23:00:49.486 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149701 for osd.8 2022-03-08T23:00:49.492 INFO:teuthology.orchestra.run.smithi042.stdout:77309411339 2022-03-08T23:00:49.526 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411340 got 77309411339 for osd.0 2022-03-08T23:00:49.617 INFO:teuthology.orchestra.run.smithi042.stdout:111669149702 2022-03-08T23:00:49.642 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149702 for osd.7 2022-03-08T23:00:49.653 INFO:teuthology.orchestra.run.smithi042.stdout:98784247816 2022-03-08T23:00:49.677 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247817 got 98784247816 for osd.2 2022-03-08T23:00:49.744 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:49.752 INFO:teuthology.orchestra.run.smithi042.stdout:115964116997 2022-03-08T23:00:49.768 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116998 for osd.4 2022-03-08T23:00:49.775 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116997 for osd.6 2022-03-08T23:00:49.808 INFO:teuthology.orchestra.run.smithi042.stdout:150323855363 2022-03-08T23:00:49.823 INFO:teuthology.orchestra.run.smithi042.stdout:150323855364 2022-03-08T23:00:49.828 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:49.830 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855364 got 150323855363 for osd.11 2022-03-08T23:00:49.846 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116998 for osd.3 2022-03-08T23:00:49.850 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855365 got 150323855364 for osd.10 2022-03-08T23:00:50.373 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.5 2022-03-08T23:00:50.478 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.1 2022-03-08T23:00:50.483 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.9 2022-03-08T23:00:50.487 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.8 2022-03-08T23:00:50.527 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.0 2022-03-08T23:00:50.643 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.7 2022-03-08T23:00:50.678 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.2 2022-03-08T23:00:50.769 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.4 2022-03-08T23:00:50.776 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.6 2022-03-08T23:00:50.832 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.11 2022-03-08T23:00:50.847 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.3 2022-03-08T23:00:50.851 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph osd last-stat-seq osd.10 2022-03-08T23:00:51.084 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:51.152 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.5 2022-03-08T23:00:51.152 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.187 INFO:teuthology.orchestra.run.smithi042.stdout:137438953477 2022-03-08T23:00:51.213 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953476 got 137438953477 for osd.9 2022-03-08T23:00:51.213 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.278 INFO:teuthology.orchestra.run.smithi042.stdout:81604378634 2022-03-08T23:00:51.278 INFO:teuthology.orchestra.run.smithi042.stdout:77309411340 2022-03-08T23:00:51.376 INFO:teuthology.orchestra.run.smithi042.stdout:111669149702 2022-03-08T23:00:51.416 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411340 got 77309411340 for osd.0 2022-03-08T23:00:51.417 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.421 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378634 got 81604378634 for osd.1 2022-03-08T23:00:51.422 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.427 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149702 for osd.8 2022-03-08T23:00:51.427 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.535 INFO:teuthology.orchestra.run.smithi042.stdout:111669149703 2022-03-08T23:00:51.536 INFO:teuthology.orchestra.run.smithi042.stdout:98784247817 2022-03-08T23:00:51.561 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149703 for osd.7 2022-03-08T23:00:51.561 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.564 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247817 got 98784247817 for osd.2 2022-03-08T23:00:51.565 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.704 INFO:teuthology.orchestra.run.smithi042.stdout:115964116999 2022-03-08T23:00:51.728 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116999 for osd.4 2022-03-08T23:00:51.728 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.761 INFO:teuthology.orchestra.run.smithi042.stdout:115964116998 2022-03-08T23:00:51.769 INFO:teuthology.orchestra.run.smithi042.stdout:115964116999 2022-03-08T23:00:51.779 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.6 2022-03-08T23:00:51.779 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.791 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116999 for osd.3 2022-03-08T23:00:51.792 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.834 INFO:teuthology.orchestra.run.smithi042.stdout:150323855364 2022-03-08T23:00:51.839 INFO:teuthology.orchestra.run.smithi042.stdout:150323855365 2022-03-08T23:00:51.857 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855364 got 150323855364 for osd.11 2022-03-08T23:00:51.858 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.861 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855365 got 150323855365 for osd.10 2022-03-08T23:00:51.861 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:51.861 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T23:00:51.862 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph pg dump --format=json 2022-03-08T23:00:52.359 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:52.360 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2022-03-08T23:00:52.376 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2022-03-08T23:00:52.036153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66984,"kb_used_data":5640,"kb_used_omap":0,"kb_used_meta":60864,"kb_avail":1124874840,"statfs":{"total":1151940427776,"available":1151871836160,"internally_reserved":0,"allocated":5775360,"data_stored":3376368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62324736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"1.0","version":"36'137","reported_seq":193,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-03-08T23:00:40.337899+0000","last_change":"2022-03-08T23:00:28.397389+0000","last_active":"2022-03-08T23:00:40.337899+0000","last_peered":"2022-03-08T23:00:40.337899+0000","last_clean":"2022-03-08T23:00:40.337899+0000","last_became_active":"2022-03-08T23:00:28.395342+0000","last_became_peered":"2022-03-08T23:00:28.395342+0000","last_unstale":"2022-03-08T23:00:40.337899+0000","last_undegraded":"2022-03-08T23:00:40.337899+0000","last_fullsized":"2022-03-08T23:00:40.337899+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:00:23.126724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:00:23.126724+0000","last_clean_scrub_stamp":"2022-03-08T23:00:23.126724+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T08:56:38.038992+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":35,"seq":150323855364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]}]},{"osd":10,"up_from":35,"seq":150323855365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0459999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3809999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.524}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3479999999999999}]}]},{"osd":9,"up_from":32,"seq":137438953477,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2429999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":6,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.905}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5840000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4430000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8490000000000002}]}]},{"osd":5,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]}]},{"osd":4,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.117}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2799999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]}]},{"osd":1,"up_from":19,"seq":81604378634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.74}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2440000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1320000000000001}]}]},{"osd":8,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.891}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1720000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2010000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.383}]}]},{"osd":0,"up_from":18,"seq":77309411341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9950000000000001}]}]},{"osd":7,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.133}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4829999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9829999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7629999999999999}]}]},{"osd":2,"up_from":23,"seq":98784247818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9580000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9830000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.101}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8759999999999999}]}]},{"osd":3,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:00:52.378 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph pg dump --format=json 2022-03-08T23:00:52.950 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:52.952 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2022-03-08T23:00:52.969 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2022-03-08T23:00:52.036153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66984,"kb_used_data":5640,"kb_used_omap":0,"kb_used_meta":60864,"kb_avail":1124874840,"statfs":{"total":1151940427776,"available":1151871836160,"internally_reserved":0,"allocated":5775360,"data_stored":3376368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62324736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"1.0","version":"36'137","reported_seq":193,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-03-08T23:00:40.337899+0000","last_change":"2022-03-08T23:00:28.397389+0000","last_active":"2022-03-08T23:00:40.337899+0000","last_peered":"2022-03-08T23:00:40.337899+0000","last_clean":"2022-03-08T23:00:40.337899+0000","last_became_active":"2022-03-08T23:00:28.395342+0000","last_became_peered":"2022-03-08T23:00:28.395342+0000","last_unstale":"2022-03-08T23:00:40.337899+0000","last_undegraded":"2022-03-08T23:00:40.337899+0000","last_fullsized":"2022-03-08T23:00:40.337899+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:00:23.126724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:00:23.126724+0000","last_clean_scrub_stamp":"2022-03-08T23:00:23.126724+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T08:56:38.038992+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":35,"seq":150323855364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]}]},{"osd":10,"up_from":35,"seq":150323855365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0459999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3809999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.524}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3479999999999999}]}]},{"osd":9,"up_from":32,"seq":137438953477,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5212,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2429999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":6,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.905}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5840000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4430000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8490000000000002}]}]},{"osd":5,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]}]},{"osd":4,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.117}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2799999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]}]},{"osd":1,"up_from":19,"seq":81604378634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.74}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2440000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1320000000000001}]}]},{"osd":8,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.891}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1720000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2010000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.383}]}]},{"osd":0,"up_from":18,"seq":77309411341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9950000000000001}]}]},{"osd":7,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.133}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4829999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9829999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7629999999999999}]}]},{"osd":2,"up_from":23,"seq":98784247818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":678060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9580000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9830000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.101}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8759999999999999}]}]},{"osd":3,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5276,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739876,"statfs":{"total":95995035648,"available":95989633024,"internally_reserved":0,"allocated":348160,"data_stored":149132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:00:52.971 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T23:00:52.971 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T23:00:52.971 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T23:00:52.972 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph health --format=json 2022-03-08T23:00:53.519 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:53.519 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T23:00:53.539 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T23:00:53.539 INFO:tasks.rook:Rook complete, yielding 2022-03-08T23:00:53.539 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:00:53.550 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph -s 2022-03-08T23:00:54.203 INFO:teuthology.orchestra.run.smithi042.stdout: cluster: 2022-03-08T23:00:54.203 INFO:teuthology.orchestra.run.smithi042.stdout: id: 20eb5159-27d7-4658-84d5-b16f4f92328a 2022-03-08T23:00:54.204 INFO:teuthology.orchestra.run.smithi042.stdout: health: HEALTH_OK 2022-03-08T23:00:54.204 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:54.204 INFO:teuthology.orchestra.run.smithi042.stdout: services: 2022-03-08T23:00:54.204 INFO:teuthology.orchestra.run.smithi042.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-08T23:00:54.204 INFO:teuthology.orchestra.run.smithi042.stdout: mgr: a(active, since 2m) 2022-03-08T23:00:54.205 INFO:teuthology.orchestra.run.smithi042.stdout: osd: 12 osds: 12 up (since 17s), 12 in (since 59s) 2022-03-08T23:00:54.205 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:54.205 INFO:teuthology.orchestra.run.smithi042.stdout: data: 2022-03-08T23:00:54.205 INFO:teuthology.orchestra.run.smithi042.stdout: pools: 1 pools, 1 pgs 2022-03-08T23:00:54.205 INFO:teuthology.orchestra.run.smithi042.stdout: objects: 2 objects, 577 KiB 2022-03-08T23:00:54.206 INFO:teuthology.orchestra.run.smithi042.stdout: usage: 65 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-08T23:00:54.206 INFO:teuthology.orchestra.run.smithi042.stdout: pgs: 1 active+clean 2022-03-08T23:00:54.206 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:54.206 INFO:teuthology.orchestra.run.smithi042.stdout: io: 2022-03-08T23:00:54.207 INFO:teuthology.orchestra.run.smithi042.stdout: client: 767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-03-08T23:00:54.207 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:00:54.223 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status 2022-03-08T23:00:54.903 INFO:teuthology.orchestra.run.smithi042.stdout:Backend: rook 2022-03-08T23:00:54.904 INFO:teuthology.orchestra.run.smithi042.stdout:Available: Yes 2022-03-08T23:00:54.923 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ps 2022-03-08T23:00:55.471 INFO:teuthology.orchestra.run.smithi042.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:00:55.471 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi042 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T23:00:55.471 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi153 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T23:00:55.471 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi178 running (68s) 0s ago 68s - - 1c5288aef625 2022-03-08T23:00:55.472 INFO:teuthology.orchestra.run.smithi042.stdout:mgr.a smithi178 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T23:00:55.472 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a smithi153 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-08T23:00:55.472 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b smithi178 running (4m) 0s ago 4m - - 1c5288aef625 2022-03-08T23:00:55.473 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c smithi042 running (4m) 0s ago 4m - - 1c5288aef625 2022-03-08T23:00:55.473 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 smithi042 running (78s) 0s ago 78s - - 1c5288aef625 2022-03-08T23:00:55.473 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1 smithi153 running (72s) 0s ago 73s - - 1c5288aef625 2022-03-08T23:00:55.473 INFO:teuthology.orchestra.run.smithi042.stdout:osd.10 smithi042 running (54s) 0s ago 55s - - 1c5288aef625 2022-03-08T23:00:55.473 INFO:teuthology.orchestra.run.smithi042.stdout:osd.11 smithi042 running (51s) 0s ago 51s - - 1c5288aef625 2022-03-08T23:00:55.474 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2 smithi153 running (70s) 0s ago 71s - - 1c5288aef625 2022-03-08T23:00:55.474 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 smithi178 running (68s) 0s ago 68s - - 1c5288aef625 2022-03-08T23:00:55.474 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4 smithi178 running (67s) 0s ago 67s - - 1c5288aef625 2022-03-08T23:00:55.474 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5 smithi178 running (65s) 0s ago 66s - - 1c5288aef625 2022-03-08T23:00:55.475 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6 smithi178 running (63s) 0s ago 64s - - 1c5288aef625 2022-03-08T23:00:55.475 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7 smithi153 running (62s) 0s ago 62s - - 1c5288aef625 2022-03-08T23:00:55.475 INFO:teuthology.orchestra.run.smithi042.stdout:osd.8 smithi153 running (60s) 0s ago 60s - - 1c5288aef625 2022-03-08T23:00:55.475 INFO:teuthology.orchestra.run.smithi042.stdout:osd.9 smithi042 running (56s) 0s ago 56s - - 1c5288aef625 2022-03-08T23:00:55.489 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls 2022-03-08T23:00:56.179 INFO:teuthology.orchestra.run.smithi042.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:00:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:crash 3/3 0s ago 2m * 2022-03-08T23:00:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T23:00:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:mon 3/3 0s ago 5m count:3 2022-03-08T23:00:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:osd 24 0s ago 79s 2022-03-08T23:00:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:00:56.203 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch device ls 2022-03-08T23:00:56.766 INFO:teuthology.orchestra.run.smithi042.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:00:56.766 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:00:56.766 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:00:56.766 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:00:56.767 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:00:56.767 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:00:56.767 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:00:56.767 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:00:56.768 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:00:56.768 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:00:56.768 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:00:56.768 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:00:56.768 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:00:56.789 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T23:00:56.802 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T23:00:56.802 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi042.front.sepia.ceph.com 2022-03-08T23:00:56.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T23:00:56.803 DEBUG:teuthology.orchestra.run.smithi042:> toolbox() { 2022-03-08T23:00:56.803 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T23:00:56.803 DEBUG:teuthology.orchestra.run.smithi042:> } 2022-03-08T23:00:56.804 DEBUG:teuthology.orchestra.run.smithi042:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:00:56.804 DEBUG:teuthology.orchestra.run.smithi042:> toolbox ceph orch osd rm 0 --force 2022-03-08T23:00:56.804 DEBUG:teuthology.orchestra.run.smithi042:> removed_pv="" 2022-03-08T23:00:56.805 DEBUG:teuthology.orchestra.run.smithi042:> while [ "$removed_pv" = "" ] 2022-03-08T23:00:56.805 DEBUG:teuthology.orchestra.run.smithi042:> do 2022-03-08T23:00:56.805 DEBUG:teuthology.orchestra.run.smithi042:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T23:00:56.805 DEBUG:teuthology.orchestra.run.smithi042:> sleep 3s 2022-03-08T23:00:56.806 DEBUG:teuthology.orchestra.run.smithi042:> done 2022-03-08T23:00:56.806 DEBUG:teuthology.orchestra.run.smithi042:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T23:00:56.806 DEBUG:teuthology.orchestra.run.smithi042:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T23:00:56.807 DEBUG:teuthology.orchestra.run.smithi042:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T23:00:56.807 DEBUG:teuthology.orchestra.run.smithi042:> zap_completion="0" 2022-03-08T23:00:56.807 DEBUG:teuthology.orchestra.run.smithi042:> while [ "$zap_completion" = "0" ] 2022-03-08T23:00:56.808 DEBUG:teuthology.orchestra.run.smithi042:> do 2022-03-08T23:00:56.808 DEBUG:teuthology.orchestra.run.smithi042:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T23:00:56.808 DEBUG:teuthology.orchestra.run.smithi042:> sleep 3s 2022-03-08T23:00:56.809 DEBUG:teuthology.orchestra.run.smithi042:> done 2022-03-08T23:00:56.809 DEBUG:teuthology.orchestra.run.smithi042:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T23:00:56.809 DEBUG:teuthology.orchestra.run.smithi042:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:00:56.809 DEBUG:teuthology.orchestra.run.smithi042:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:00:56.810 DEBUG:teuthology.orchestra.run.smithi042:> num_osd="0" 2022-03-08T23:00:56.810 DEBUG:teuthology.orchestra.run.smithi042:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T23:00:56.810 DEBUG:teuthology.orchestra.run.smithi042:> do 2022-03-08T23:00:56.811 DEBUG:teuthology.orchestra.run.smithi042:> echo "waiting for osd to come back up" 2022-03-08T23:00:56.811 DEBUG:teuthology.orchestra.run.smithi042:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:00:56.811 DEBUG:teuthology.orchestra.run.smithi042:> sleep 30s 2022-03-08T23:00:56.811 DEBUG:teuthology.orchestra.run.smithi042:> done 2022-03-08T23:00:56.812 DEBUG:teuthology.orchestra.run.smithi042:> ' 2022-03-08T23:00:56.828 INFO:teuthology.orchestra.run.smithi042.stderr:++ toolbox ceph osd stat 2022-03-08T23:00:56.828 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:00:56.829 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f3 -d ' ' 2022-03-08T23:00:58.896 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:59.378 INFO:teuthology.orchestra.run.smithi042.stderr:+ orig_num_osd=12 2022-03-08T23:00:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T23:00:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T23:00:59.454 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:00.072 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.074283 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:01:00.072 INFO:tasks.rook.operator.smithi042.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:01:00.073 INFO:tasks.rook.operator.smithi042.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:01:00.073 INFO:tasks.rook.operator.smithi042.stdout:   ... // 3 identical fields 2022-03-08T23:01:00.073 INFO:tasks.rook.operator.smithi042.stdout:   Config: nil, 2022-03-08T23:01:00.074 INFO:tasks.rook.operator.smithi042.stdout:   Selection: {}, 2022-03-08T23:01:00.074 INFO:tasks.rook.operator.smithi042.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:01:00.074 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi042-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi042-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:00.075 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi042-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi042-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:00.075 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi042-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi042-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:00.075 INFO:tasks.rook.operator.smithi042.stdout:-  { 2022-03-08T23:01:00.076 INFO:tasks.rook.operator.smithi042.stdout:-  Name: "smithi042-lv-4", 2022-03-08T23:01:00.076 INFO:tasks.rook.operator.smithi042.stdout:-  Count: 1, 2022-03-08T23:01:00.077 INFO:tasks.rook.operator.smithi042.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:01:00.077 INFO:tasks.rook.operator.smithi042.stdout:-  }, 2022-03-08T23:01:00.077 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi153-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi153-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:00.078 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi153-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi153-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:00.078 INFO:tasks.rook.operator.smithi042.stdout:   ... // 6 identical elements 2022-03-08T23:01:00.078 INFO:tasks.rook.operator.smithi042.stdout:   }, 2022-03-08T23:01:00.079 INFO:tasks.rook.operator.smithi042.stdout:   }, 2022-03-08T23:01:00.079 INFO:tasks.rook.operator.smithi042.stdout:   Annotations: nil, 2022-03-08T23:01:00.080 INFO:tasks.rook.operator.smithi042.stdout:   Labels: nil, 2022-03-08T23:01:00.080 INFO:tasks.rook.operator.smithi042.stdout:   ... // 20 identical fields 2022-03-08T23:01:00.080 INFO:tasks.rook.operator.smithi042.stdout:  } 2022-03-08T23:01:00.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.074367 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:01:00.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.074719 I | operator: successfully started the controller-runtime manager 2022-03-08T23:01:00.081 INFO:tasks.rook.operator.smithi042.stdout:I0308 23:01:00.074740 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:01:00.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.080995 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:01:00.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.081019 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:01:00.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.081051 I | operator: setting up schemes 2022-03-08T23:01:00.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.083700 I | operator: setting up the controller-runtime manager 2022-03-08T23:01:00.283 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.287574 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:01:00.290 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290701 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:01:00.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290748 I | ceph-cluster-controller: successfully started 2022-03-08T23:01:00.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290822 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:01:00.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290855 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:01:00.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290882 I | ceph-block-pool-controller: successfully started 2022-03-08T23:01:00.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290908 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:01:00.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290940 I | ceph-object-realm-controller: successfully started 2022-03-08T23:01:00.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290964 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:01:00.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.290982 I | ceph-object-zone-controller: successfully started 2022-03-08T23:01:00.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291163 I | ceph-object-controller: successfully started 2022-03-08T23:01:00.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291213 I | ceph-file-controller: successfully started 2022-03-08T23:01:00.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291267 I | ceph-nfs-controller: successfully started 2022-03-08T23:01:00.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291318 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:01:00.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291362 I | ceph-client-controller: successfully started 2022-03-08T23:01:00.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291392 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:01:00.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291443 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:01:00.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291475 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:01:00.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291511 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:01:00.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291542 I | ceph-bucket-topic: successfully started 2022-03-08T23:01:00.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291568 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:00.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291592 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:00.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.291615 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:01:00.296 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.294940 I | operator: starting the controller-runtime manager 2022-03-08T23:01:00.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.396338 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:01:00.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.396393 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:01:00.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.396412 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:01:00.396 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.398261 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:00.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.502312 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:01:00.503 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.504705 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:00.503 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.504773 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:01:00.503 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.504787 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:00.504 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.506761 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:01:00.504 INFO:tasks.rook.operator.smithi042.stdout:I0308 23:01:00.506843 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:00.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.534858 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:00.574 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.578321 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:01:00.662 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.666477 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:01:00.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701060 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:01:00.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701087 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:01:00.698 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701096 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:01:00.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701104 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:01:00.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701112 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:01:00.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701121 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:01:00.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701131 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:01:00.700 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701146 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:01:00.700 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701163 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:01:00.700 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701178 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:01:00.701 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701192 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:01:00.701 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701209 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:01:00.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701225 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:01:00.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701237 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:01:00.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701250 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:01:00.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701265 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:01:00.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701352 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:00.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:00.701380 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:04.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:04.348291 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:01:04.804 INFO:teuthology.orchestra.run.smithi042.stdout:Success 2022-03-08T23:01:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:+ removed_pv= 2022-03-08T23:01:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' '' = '' ']' 2022-03-08T23:01:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl get pv 2022-03-08T23:01:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:++ grep Released 2022-03-08T23:01:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f1 -d ' ' 2022-03-08T23:01:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:+ removed_pv=smithi042-lv-4 2022-03-08T23:01:04.904 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 3s 2022-03-08T23:01:07.905 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' smithi042-lv-4 = '' ']' 2022-03-08T23:01:07.905 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl get pv smithi042-lv-4 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:01:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:+ target_path=/dev/vg_nvme/lv_4 2022-03-08T23:01:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:++ echo smithi042-lv-4 2022-03-08T23:01:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f1 -d - 2022-03-08T23:01:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:+ host=smithi042 2022-03-08T23:01:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:+ toolbox ceph orch device zap smithi042 /dev/vg_nvme/lv_4 --force 2022-03-08T23:01:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi042 /dev/vg_nvme/lv_4 --force 2022-03-08T23:01:08.084 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:08.534 INFO:teuthology.orchestra.run.smithi042.stdout:/dev/vg_nvme/lv_4 on smithi042 zapped 2022-03-08T23:01:08.557 INFO:teuthology.orchestra.run.smithi042.stderr:+ zap_completion=0 2022-03-08T23:01:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:01:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:01:08.634 INFO:teuthology.orchestra.run.smithi042.stderr:+ zap_completion= 2022-03-08T23:01:08.634 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 3s 2022-03-08T23:01:10.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118068 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:01:10.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118115 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:01:10.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118127 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:01:10.119 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118135 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:01:10.119 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118146 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:01:10.119 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118156 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:01:10.119 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118163 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:01:10.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118173 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:01:10.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118181 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:01:10.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118189 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:01:10.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118199 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:01:10.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118218 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:01:10.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118229 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:01:10.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118240 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:01:10.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118264 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:01:10.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118276 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:01:10.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118292 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:01:10.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118303 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:01:10.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118318 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:01:10.123 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118331 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:01:10.123 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118379 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:10.123 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118434 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:01:10.123 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118459 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:01:10.124 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118477 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:01:10.124 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118484 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:01:10.124 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118491 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:01:10.124 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118499 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:01:10.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.118509 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:01:10.182 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.185065 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:01:10.189 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.188765 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:10.194 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197164 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:10.194 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197182 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.195 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197185 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:10.195 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197187 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.195 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197190 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:10.195 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197192 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.197195 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:01:10.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.301460 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:10.301 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.301641 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:10.393 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.396385 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:10.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.396403 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.396406 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:01:10.539 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.541972 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:01:10.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.754388 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:10.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.754437 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.754450 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:01:10.773 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.775633 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:01:10.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.826923 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:10.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.826943 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:10.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.826949 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:01:10.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:10.991601 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:01:11.112 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.060327 I | op-mon: start running mons 2022-03-08T23:01:11.163 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.165165 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' '' = 0 ']' 2022-03-08T23:01:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:+ kubectl patch pv smithi042-lv-4 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:01:11.710 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.712565 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:01:11.906 INFO:teuthology.orchestra.run.smithi042.stdout:persistentvolume/smithi042-lv-4 patched 2022-03-08T23:01:11.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.908536 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:01:11.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.908567 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:01:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:01:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:01:11.963 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:11.966248 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.111.142.38:6789","10.100.208.21:6789","10.110.240.83:6789"]}] data:c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T23:01:12.177 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:12.179699 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:01:12.248 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:12.505 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:12.507955 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:12.506 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:12.508152 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:12.988 INFO:teuthology.orchestra.run.smithi042.stdout:['Success'] 2022-03-08T23:01:12.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:12.998059 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:01:12.997 INFO:tasks.rook.operator.smithi042.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:01:12.997 INFO:tasks.rook.operator.smithi042.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:01:12.997 INFO:tasks.rook.operator.smithi042.stdout:   ... // 3 identical fields 2022-03-08T23:01:12.998 INFO:tasks.rook.operator.smithi042.stdout:   Config: nil, 2022-03-08T23:01:12.998 INFO:tasks.rook.operator.smithi042.stdout:   Selection: {}, 2022-03-08T23:01:12.998 INFO:tasks.rook.operator.smithi042.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:01:12.998 INFO:tasks.rook.operator.smithi042.stdout:   ... // 9 identical elements 2022-03-08T23:01:12.999 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi178-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi178-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:12.999 INFO:tasks.rook.operator.smithi042.stdout:   {Name: "smithi178-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi178-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:01:12.999 INFO:tasks.rook.operator.smithi042.stdout:+  { 2022-03-08T23:01:12.999 INFO:tasks.rook.operator.smithi042.stdout:+  Name: "smithi042-lv-4", 2022-03-08T23:01:13.000 INFO:tasks.rook.operator.smithi042.stdout:+  Count: 1, 2022-03-08T23:01:13.000 INFO:tasks.rook.operator.smithi042.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:01:13.000 INFO:tasks.rook.operator.smithi042.stdout:+  }, 2022-03-08T23:01:13.000 INFO:tasks.rook.operator.smithi042.stdout:   }, 2022-03-08T23:01:13.000 INFO:tasks.rook.operator.smithi042.stdout:   }, 2022-03-08T23:01:13.001 INFO:tasks.rook.operator.smithi042.stdout:   Annotations: nil, 2022-03-08T23:01:13.001 INFO:tasks.rook.operator.smithi042.stdout:   Labels: nil, 2022-03-08T23:01:13.001 INFO:tasks.rook.operator.smithi042.stdout:   ... // 20 identical fields 2022-03-08T23:01:13.002 INFO:tasks.rook.operator.smithi042.stdout:  } 2022-03-08T23:01:13.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:12.998160 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:01:13.002 INFO:tasks.rook.operator.smithi042.stdout:I0308 23:01:12.998677 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:01:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:01:13.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.290164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to initialize ceph cluster info: failed to save mon keyring secret: failed to get secret for rook-ceph-mons-keyring: context canceled 2022-03-08T23:01:13.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.290326 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:13.289 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.290487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to load cluster info: failed to get mon secrets: context canceled 2022-03-08T23:01:13.289 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.290558 I | operator: successfully started the controller-runtime manager 2022-03-08T23:01:13.488 INFO:teuthology.orchestra.run.smithi042.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:01:13.488 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.489770 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:01:13.489 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.489801 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:01:13.489 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.489835 I | operator: setting up schemes 2022-03-08T23:01:13.492 INFO:teuthology.orchestra.run.smithi042.stderr:+ num_osd=0 2022-03-08T23:01:13.492 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 0 '!=' 12 ']' 2022-03-08T23:01:13.493 INFO:teuthology.orchestra.run.smithi042.stdout:waiting for osd to come back up 2022-03-08T23:01:13.493 INFO:teuthology.orchestra.run.smithi042.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:01:13.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.495333 I | operator: setting up the controller-runtime manager 2022-03-08T23:01:13.494 INFO:teuthology.orchestra.run.smithi042.stderr:++ toolbox ceph osd stat 2022-03-08T23:01:13.495 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f3 -d ' ' 2022-03-08T23:01:13.495 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:01:13.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.697976 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:01:13.721 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723577 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:01:13.721 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723624 I | ceph-cluster-controller: successfully started 2022-03-08T23:01:13.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723684 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:01:13.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723714 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:01:13.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723741 I | ceph-block-pool-controller: successfully started 2022-03-08T23:01:13.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723763 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:01:13.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723785 I | ceph-object-realm-controller: successfully started 2022-03-08T23:01:13.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723807 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:01:13.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723830 I | ceph-object-zone-controller: successfully started 2022-03-08T23:01:13.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.723998 I | ceph-object-controller: successfully started 2022-03-08T23:01:13.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724046 I | ceph-file-controller: successfully started 2022-03-08T23:01:13.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724083 I | ceph-nfs-controller: successfully started 2022-03-08T23:01:13.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724129 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:01:13.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724188 I | ceph-client-controller: successfully started 2022-03-08T23:01:13.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724231 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:01:13.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724282 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:01:13.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724305 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:01:13.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724327 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:01:13.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724347 I | ceph-bucket-topic: successfully started 2022-03-08T23:01:13.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724363 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:13.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724380 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:13.728 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.724398 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:01:13.731 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.725651 I | operator: starting the controller-runtime manager 2022-03-08T23:01:13.737 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:13.833 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.834439 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:01:13.834 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.834530 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:01:13.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.834578 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:01:13.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.834765 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:13.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:13.837374 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:01:14.124 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.126602 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:14.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.126716 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:01:14.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.126742 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:14.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.129772 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:01:14.129 INFO:tasks.rook.operator.smithi042.stdout:I0308 23:01:14.129896 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:01:14.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.292868 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:+ num_osd=11 2022-03-08T23:01:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 30s 2022-03-08T23:01:14.637 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.638898 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:01:14.659 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.661179 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:01:14.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903537 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:01:14.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903565 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:01:14.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903574 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:01:14.904 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903581 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:01:14.904 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903588 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:01:14.904 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903595 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:01:14.904 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903604 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:01:14.904 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903615 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:01:14.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903629 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:01:14.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903645 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:01:14.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903656 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:01:14.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903673 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:01:14.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903689 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:01:14.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903701 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:01:14.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903713 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:01:14.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903728 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:01:14.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903839 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:14.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:14.903860 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:22.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.477729 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:01:22.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.477746 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:22.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.477753 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:01:22.486 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.487557 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:22.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.609172 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:22.649 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:22.648427 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:23.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.112464 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:01:23.188 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.189586 I | op-mon: start running mons 2022-03-08T23:01:23.214 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.215399 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:01:23.237 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.238604 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.111.142.38:6789","10.100.208.21:6789","10.110.240.83:6789"]}] data:a=10.100.208.21:6789,b=10.110.240.83:6789,c=10.111.142.38:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T23:01:23.596 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.596763 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:23.596 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:23.597075 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:24.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801601 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:01:24.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801631 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:01:24.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801640 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:01:24.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801644 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:01:24.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801649 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:01:24.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801653 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:01:24.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801657 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:01:24.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801660 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:01:24.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801665 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:01:24.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801669 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:01:24.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801676 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:01:24.806 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801680 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:01:24.806 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801684 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:01:24.806 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801689 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:01:24.806 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801694 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:01:24.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801698 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:01:24.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801703 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:01:24.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801712 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:01:24.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801719 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:01:24.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801724 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:01:24.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801729 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:01:24.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801734 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:01:24.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801738 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:01:24.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801747 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:01:24.809 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:24.801757 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:01:25.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.197456 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:01:25.207 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203426 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:25.208 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203522 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:25.208 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203547 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:25.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203570 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:25.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203592 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:25.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203612 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:25.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.203632 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:01:25.268 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.269083 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:25.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.269115 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:25.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.269124 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:01:25.311 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:25.312068 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:01:26.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.196710 I | op-mon: targeting the mon count 3 2022-03-08T23:01:26.202 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.203364 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:26.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.398548 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:01:26.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.398580 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:01:26.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.398590 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:01:26.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.448051 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:26.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.448079 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:26.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.448087 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:01:26.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.492841 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:01:26.559 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.559516 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:26.559 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.559555 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:26.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.994620 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:26.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.994650 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:26.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:26.999236 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:01:27.003 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.004392 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:01:27.003 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.004406 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:01:27.010 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.009560 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:01:27.510 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.510502 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:27.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.510530 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:27.876 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.876783 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:27.877 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:27.876813 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:01:28.286 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:28.285954 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:01:28.286 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:28.285992 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:01:28.476 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:28.476710 I | op-mon: mon "c" endpoint is [v2:10.111.142.38:3300,v1:10.111.142.38:6789] 2022-03-08T23:01:28.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:28.663023 I | op-mon: mon "a" endpoint is [v2:10.100.208.21:3300,v1:10.100.208.21:6789] 2022-03-08T23:01:29.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:29.399396 I | op-mon: mon "b" endpoint is [v2:10.110.240.83:3300,v1:10.110.240.83:6789] 2022-03-08T23:01:29.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:29.997725 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.240.83:6789","10.111.142.38:6789","10.100.208.21:6789"]}] data:c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 mapping:{"node":{"a":{"Name":"smithi153","Hostname":"smithi153","Address":"172.21.15.153"},"b":{"Name":"smithi178","Hostname":"smithi178","Address":"172.21.15.178"},"c":{"Name":"smithi042","Hostname":"smithi042","Address":"172.21.15.42"}}} maxMonId:2] 2022-03-08T23:01:30.596 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:30.596557 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:30.601 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:30.596834 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:31.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:31.020754 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:01:31.028 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:31.028730 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:01:31.028 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:31.028757 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:01:31.604 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:31.603837 I | op-mon: mons running: [c a b] 2022-03-08T23:01:32.014 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.014055 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:01:32.022 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.022163 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:01:32.032 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.032135 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:01:32.032 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.032160 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:01:32.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.206005 I | op-mon: mons running: [c a b] 2022-03-08T23:01:32.607 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.607463 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:01:32.616 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.616538 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:01:32.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.627233 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:01:32.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.627249 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:01:32.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:32.806862 I | op-mon: mons running: [c a b] 2022-03-08T23:01:33.251 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:33.250710 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:01:33.251 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:33.250733 I | op-mon: mons created: 3 2022-03-08T23:01:33.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:33.649615 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:01:33.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:33.689636 I | op-mon: mons running: [c a b] 2022-03-08T23:01:34.096 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:34.096086 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:01:34.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:34.096132 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:01:34.523 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:34.523178 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:01:34.932 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:34.931436 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:01:35.371 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:35.370367 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:01:35.827 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:35.826354 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:01:35.827 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:35.826386 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:01:36.265 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:36.264436 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:01:36.680 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:36.679072 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:01:36.680 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:36.679118 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:01:36.680 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:36.679131 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:01:37.116 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.115651 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:01:37.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.173787 I | op-mgr: start running mgr 2022-03-08T23:01:37.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.173832 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:01:37.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.689416 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:01:37.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.747436 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:01:37.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.751601 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:01:37.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.865385 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:01:37.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.865716 I | op-mgr: successful modules: balancer 2022-03-08T23:01:37.879 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.878793 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:01:37.884 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.878998 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:01:37.964 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.962027 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:01:37.992 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990666 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi042-lv-1" 2022-03-08T23:01:37.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990764 I | op-osd: OSD PVC "smithi042-lv-1-data-09lqvf" already exists 2022-03-08T23:01:37.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990787 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi042-lv-2" 2022-03-08T23:01:37.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990801 I | op-osd: OSD PVC "smithi042-lv-2-data-0khnqg" already exists 2022-03-08T23:01:37.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990814 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi042-lv-3" 2022-03-08T23:01:37.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990828 I | op-osd: OSD PVC "smithi042-lv-3-data-084p6f" already exists 2022-03-08T23:01:37.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990841 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi153-lv-1" 2022-03-08T23:01:37.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990857 I | op-osd: OSD PVC "smithi153-lv-1-data-0vtz4j" already exists 2022-03-08T23:01:37.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990868 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi153-lv-2" 2022-03-08T23:01:37.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990881 I | op-osd: OSD PVC "smithi153-lv-2-data-0qw442" already exists 2022-03-08T23:01:37.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990896 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi153-lv-3" 2022-03-08T23:01:37.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990910 I | op-osd: OSD PVC "smithi153-lv-3-data-0bj679" already exists 2022-03-08T23:01:37.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990920 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi153-lv-4" 2022-03-08T23:01:37.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990938 I | op-osd: OSD PVC "smithi153-lv-4-data-0h87v5" already exists 2022-03-08T23:01:37.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990946 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi178-lv-1" 2022-03-08T23:01:37.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990959 I | op-osd: OSD PVC "smithi178-lv-1-data-0b2hpx" already exists 2022-03-08T23:01:37.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990967 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi178-lv-2" 2022-03-08T23:01:38.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990977 I | op-osd: OSD PVC "smithi178-lv-2-data-0nv8kw" already exists 2022-03-08T23:01:38.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990988 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi178-lv-3" 2022-03-08T23:01:38.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.990999 I | op-osd: OSD PVC "smithi178-lv-3-data-0sf4l2" already exists 2022-03-08T23:01:38.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.991007 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi178-lv-4" 2022-03-08T23:01:38.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.991024 I | op-osd: OSD PVC "smithi178-lv-4-data-0scdxj" already exists 2022-03-08T23:01:38.001 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:37.991033 I | op-osd: creating 1 new PVCs for device set "smithi042-lv-4" 2022-03-08T23:01:38.192 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.192144 I | op-osd: successfully provisioned PVC "smithi042-lv-4-data-0d9m9m" 2022-03-08T23:01:38.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227251 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-1-data-09lqvf" 2022-03-08T23:01:38.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227269 I | op-osd: skipping OSD prepare job creation for PVC "smithi042-lv-1-data-09lqvf" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227273 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-2-data-0khnqg" 2022-03-08T23:01:38.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227275 I | op-osd: skipping OSD prepare job creation for PVC "smithi042-lv-2-data-0khnqg" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227278 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-3-data-084p6f" 2022-03-08T23:01:38.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227281 I | op-osd: skipping OSD prepare job creation for PVC "smithi042-lv-3-data-084p6f" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227284 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-1-data-0vtz4j" 2022-03-08T23:01:38.232 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227287 I | op-osd: skipping OSD prepare job creation for PVC "smithi153-lv-1-data-0vtz4j" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.232 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227289 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-2-data-0qw442" 2022-03-08T23:01:38.232 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227292 I | op-osd: skipping OSD prepare job creation for PVC "smithi153-lv-2-data-0qw442" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.232 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227295 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-3-data-0bj679" 2022-03-08T23:01:38.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227297 I | op-osd: skipping OSD prepare job creation for PVC "smithi153-lv-3-data-0bj679" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227300 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-4-data-0h87v5" 2022-03-08T23:01:38.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227303 I | op-osd: skipping OSD prepare job creation for PVC "smithi153-lv-4-data-0h87v5" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227306 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-1-data-0b2hpx" 2022-03-08T23:01:38.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227311 I | op-osd: skipping OSD prepare job creation for PVC "smithi178-lv-1-data-0b2hpx" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227315 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-2-data-0nv8kw" 2022-03-08T23:01:38.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227318 I | op-osd: skipping OSD prepare job creation for PVC "smithi178-lv-2-data-0nv8kw" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227321 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-3-data-0sf4l2" 2022-03-08T23:01:38.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227324 I | op-osd: skipping OSD prepare job creation for PVC "smithi178-lv-3-data-0sf4l2" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227327 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-4-data-0scdxj" 2022-03-08T23:01:38.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227329 I | op-osd: skipping OSD prepare job creation for PVC "smithi178-lv-4-data-0scdxj" because OSD daemon using the PVC already exists 2022-03-08T23:01:38.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.227332 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-4-data-0d9m9m" 2022-03-08T23:01:38.606 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.605587 I | op-osd: started OSD provisioning job for PVC "smithi042-lv-4-data-0d9m9m" 2022-03-08T23:01:38.607 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.605615 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:01:38.607 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.605622 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:01:38.790 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.789738 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0d9m9m is "starting" 2022-03-08T23:01:38.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.890837 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:38.896 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.896095 I | op-osd: updating OSD 1 on PVC "smithi153-lv-1-data-0vtz4j" 2022-03-08T23:01:38.896 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.896118 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-1-data-0vtz4j" 2022-03-08T23:01:38.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.954516 I | op-mgr: successful modules: prometheus 2022-03-08T23:01:38.959 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.955508 I | op-mgr: successful modules: dashboard 2022-03-08T23:01:38.960 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:38.957095 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:01:39.074 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.073489 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:39.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.081142 I | op-osd: updating OSD 10 on PVC "smithi042-lv-3-data-084p6f" 2022-03-08T23:01:39.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.081181 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-3-data-084p6f" 2022-03-08T23:01:39.285 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.284043 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:39.290 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.290266 I | op-osd: updating OSD 11 on PVC "smithi042-lv-2-data-0khnqg" 2022-03-08T23:01:39.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.290292 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-2-data-0khnqg" 2022-03-08T23:01:39.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.461878 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:39.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.497519 I | op-osd: updating OSD 2 on PVC "smithi153-lv-2-data-0qw442" 2022-03-08T23:01:39.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:39.497552 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-2-data-0qw442" 2022-03-08T23:01:40.202 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:40.201430 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:40.298 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:40.297838 I | op-osd: updating OSD 3 on PVC "smithi178-lv-3-data-0sf4l2" 2022-03-08T23:01:40.299 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:40.297875 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-3-data-0sf4l2" 2022-03-08T23:01:40.357 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:40.357079 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:01:41.006 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.005053 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:41.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.096554 I | op-osd: updating OSD 4 on PVC "smithi178-lv-2-data-0nv8kw" 2022-03-08T23:01:41.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.096595 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-2-data-0nv8kw" 2022-03-08T23:01:41.807 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.806452 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:41.898 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.897687 I | op-osd: updating OSD 5 on PVC "smithi178-lv-1-data-0b2hpx" 2022-03-08T23:01:41.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:41.897726 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-1-data-0b2hpx" 2022-03-08T23:01:42.743 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:42.741806 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:42.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:42.824617 I | op-osd: updating OSD 6 on PVC "smithi178-lv-4-data-0scdxj" 2022-03-08T23:01:42.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:42.824655 I | op-osd: OSD will have its main bluestore block on "smithi178-lv-4-data-0scdxj" 2022-03-08T23:01:43.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:43.403574 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:43.501 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:43.500281 I | op-osd: updating OSD 7 on PVC "smithi153-lv-3-data-0bj679" 2022-03-08T23:01:43.501 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:43.500329 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-3-data-0bj679" 2022-03-08T23:01:44.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:44.203473 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:44.299 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:44.298835 I | op-osd: updating OSD 8 on PVC "smithi153-lv-4-data-0h87v5" 2022-03-08T23:01:44.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:44.298870 I | op-osd: OSD will have its main bluestore block on "smithi153-lv-4-data-0h87v5" 2022-03-08T23:01:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:01:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:01:44.337 INFO:teuthology.orchestra.run.smithi042.stdout:waiting for osd to come back up 2022-03-08T23:01:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:++ toolbox ceph osd stat 2022-03-08T23:01:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:01:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f3 -d ' ' 2022-03-08T23:01:44.413 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:+ num_osd=11 2022-03-08T23:01:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 30s 2022-03-08T23:01:45.027 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:45.025832 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:01:45.100 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:45.098892 I | op-osd: updating OSD 9 on PVC "smithi042-lv-1-data-09lqvf" 2022-03-08T23:01:45.100 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:45.098927 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-1-data-09lqvf" 2022-03-08T23:01:49.624 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:49.623340 I | op-osd: OSD orchestration status for node smithi042-lv-4-data-0d9m9m is "orchestrating" 2022-03-08T23:01:50.022 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:50.020920 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0d9m9m is "orchestrating" 2022-03-08T23:01:56.172 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:56.170793 I | op-osd: OSD orchestration status for PVC smithi042-lv-4-data-0d9m9m is "completed" 2022-03-08T23:01:56.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:56.170831 I | op-osd: creating OSD 0 on PVC "smithi042-lv-4-data-0d9m9m" 2022-03-08T23:01:56.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:56.170851 I | op-osd: OSD will have its main bluestore block on "smithi042-lv-4-data-0d9m9m" 2022-03-08T23:01:57.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:57.260607 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:01:57.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:01:57.260644 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:02:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:14.943 INFO:teuthology.orchestra.run.smithi042.stdout:waiting for osd to come back up 2022-03-08T23:02:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:15.492 INFO:teuthology.orchestra.run.smithi042.stderr:+ num_osd=11 2022-03-08T23:02:15.492 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 30s 2022-03-08T23:02:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:02:45.495 INFO:teuthology.orchestra.run.smithi042.stdout:waiting for osd to come back up 2022-03-08T23:02:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:+ num_osd=12 2022-03-08T23:02:46.065 INFO:teuthology.orchestra.run.smithi042.stderr:+ sleep 30s 2022-03-08T23:03:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:+ '[' 12 '!=' 12 ']' 2022-03-08T23:03:16.068 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:03:16.082 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch status 2022-03-08T23:03:16.797 INFO:teuthology.orchestra.run.smithi042.stdout:Backend: rook 2022-03-08T23:03:16.798 INFO:teuthology.orchestra.run.smithi042.stdout:Available: Yes 2022-03-08T23:03:16.821 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ps 2022-03-08T23:03:17.398 INFO:teuthology.orchestra.run.smithi042.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:03:17.398 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi042 running (4m) 0s ago 4m - - 1c5288aef625 2022-03-08T23:03:17.398 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi153 running (4m) 0s ago 4m - - 1c5288aef625 2022-03-08T23:03:17.399 INFO:teuthology.orchestra.run.smithi042.stdout:crashcollector.crash smithi178 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.399 INFO:teuthology.orchestra.run.smithi042.stdout:mgr.a smithi178 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-08T23:03:17.399 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a smithi153 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:03:17.399 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b smithi178 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:03:17.400 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c smithi042 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-08T23:03:17.400 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 smithi042 running (81s) 0s ago 81s - - 1c5288aef625 2022-03-08T23:03:17.400 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1 smithi153 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.400 INFO:teuthology.orchestra.run.smithi042.stdout:osd.10 smithi042 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.400 INFO:teuthology.orchestra.run.smithi042.stdout:osd.11 smithi042 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.401 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2 smithi153 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.401 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 smithi178 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.401 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4 smithi178 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.401 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5 smithi178 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.402 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6 smithi178 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.402 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7 smithi153 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.402 INFO:teuthology.orchestra.run.smithi042.stdout:osd.8 smithi153 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.402 INFO:teuthology.orchestra.run.smithi042.stdout:osd.9 smithi042 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T23:03:17.419 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls 2022-03-08T23:03:18.178 INFO:teuthology.orchestra.run.smithi042.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:03:18.179 INFO:teuthology.orchestra.run.smithi042.stdout:crash 3/3 0s ago 5m * 2022-03-08T23:03:18.179 INFO:teuthology.orchestra.run.smithi042.stdout:mgr 1/1 0s ago 5m count:1 2022-03-08T23:03:18.179 INFO:teuthology.orchestra.run.smithi042.stdout:mon 3/3 0s ago 7m count:3 2022-03-08T23:03:18.179 INFO:teuthology.orchestra.run.smithi042.stdout:osd 24 0s ago 3m 2022-03-08T23:03:18.180 INFO:teuthology.orchestra.run.smithi042.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:03:18.201 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch host ls 2022-03-08T23:03:18.741 INFO:teuthology.orchestra.run.smithi042.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:03:18.741 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 172.21.15.42/smithi042 2022-03-08T23:03:18.742 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 172.21.15.153/smithi153 2022-03-08T23:03:18.742 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 172.21.15.178/smithi178 2022-03-08T23:03:18.742 INFO:teuthology.orchestra.run.smithi042.stdout:3 hosts in cluster 2022-03-08T23:03:18.762 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch device ls 2022-03-08T23:03:19.335 INFO:teuthology.orchestra.run.smithi042.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:03:19.335 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:03:19.336 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:03:19.336 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:03:19.336 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:03:19.336 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:03:19.336 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:03:19.337 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:03:19.337 INFO:teuthology.orchestra.run.smithi042.stdout:smithi153 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:03:19.337 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:03:19.337 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:03:19.338 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:03:19.338 INFO:teuthology.orchestra.run.smithi042.stdout:smithi178 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:03:19.358 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch apply rgw foo 2022-03-08T23:03:19.921 INFO:teuthology.orchestra.run.smithi042.stdout:Created 2022-03-08T23:03:19.923 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:19.921682 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:03:19.944 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch apply mds foo 2022-03-08T23:03:20.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.012205 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:20.014 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.012276 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:03:20.507 INFO:teuthology.orchestra.run.smithi042.stdout:Created 2022-03-08T23:03:20.510 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.509518 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:03:20.530 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch apply rbd-mirror 2022-03-08T23:03:20.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.539455 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:20.545 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.544462 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:20.546 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:20.544533 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:03:21.165 INFO:teuthology.orchestra.run.smithi042.stdout:Success 2022-03-08T23:03:21.185 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch apply nfs foo --port 12777 2022-03-08T23:03:21.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:21.203209 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:21.205 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:21.203283 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:03:25.247 INFO:teuthology.orchestra.run.smithi042.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:03:25.247 INFO:teuthology.orchestra.run.smithi042.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:03:25.270 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:03:25.282 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:03:25.642 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:25.640513 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:03:25.925 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:03:25.926 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:03:25.757853Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:03:25.757853Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:25.757853Z", "running": 0, "size": 1}}] 2022-03-08T23:03:25.946 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch rm mds.foo 2022-03-08T23:03:26.656 INFO:teuthology.orchestra.run.smithi042.stdout:Removed cephfilesystems/foo 2022-03-08T23:03:26.678 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch rm rbd-mirror 2022-03-08T23:03:27.041 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:27.039498 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:03:27.137 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:27.135384 I | ceph-object-controller: ceph object store gateway service running at 10.103.39.174 2022-03-08T23:03:27.137 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:27.135422 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:03:27.271 INFO:teuthology.orchestra.run.smithi042.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:03:27.291 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch rm rgw.foo 2022-03-08T23:03:27.488 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:27.486269 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:03:28.058 INFO:teuthology.orchestra.run.smithi042.stdout:Removed cephobjectstores/foo 2022-03-08T23:03:28.083 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:03:29.019 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:03:29.020 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:03:28.827101Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:28.827101Z", "running": 0, "size": 1}}] 2022-03-08T23:03:29.324 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:29.322952 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:03:29.346 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:29.344759 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:03:29.890 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:29.888429 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T23:03:29.890 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:29.888456 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T23:03:29.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:29.888466 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T23:03:31.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:31.234613 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T23:03:31.413 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:31.389748 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T23:03:31.593 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:31.592203 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi153": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:32.534 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:32.531979 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:03:32.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:32.963533 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:03:34.816 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:34.815300 I | op-mds: setting mds config flags 2022-03-08T23:03:34.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:34.815338 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:03:35.066 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:35.064587 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-08T23:03:35.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:35.722516 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:03:35.740 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:35.738947 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:03:36.063 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:36.061584 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi178": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:36.299 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:36.295422 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-08T23:03:36.335 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:36.333826 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi178": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:36.340 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:36.339668 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:03:36.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:36.342077 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-08T23:03:37.506 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:37.504638 I | op-mds: setting mds config flags 2022-03-08T23:03:37.506 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:37.504659 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:03:37.786 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:37.783912 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:03:38.373 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:38.371458 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:03:38.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:38.806071 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-08T23:03:38.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:38.835995 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:39.044 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:03:39.933 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:03:39.933 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:03:39.790484Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:39.790484Z", "running": 0, "size": 1}}] 2022-03-08T23:03:40.521 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:40.519431 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:03:43.965 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:43.962899 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:43.953+0000 7f1b58123380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:43.965 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:43.953+0000 7f1b58123380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:43.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:43.953+0000 7f1b58123380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:43.966 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:43.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:43.986810 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:44.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:44.946059 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:44.937+0000 7f799df3e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:44.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:44.937+0000 7f799df3e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:44.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:44.937+0000 7f799df3e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:44.949 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:45.015 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:45.013558 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:46.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:46.166105 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-08T23:03:46.407 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:46.405051 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:46.397+0000 7f3aab602380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:46.407 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:46.397+0000 7f3aab602380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:46.408 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:46.397+0000 7f3aab602380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:46.408 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:46.420 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:46.419611 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:47.799 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:47.690565 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:47.681+0000 7f19761f7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:47.800 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:47.681+0000 7f19761f7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:47.800 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:47.681+0000 7f19761f7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:47.800 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:47.801 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:47.728482 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:48.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:48.667755 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:48.661+0000 7f81cdaea380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:48.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:48.661+0000 7f81cdaea380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:48.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:48.661+0000 7f81cdaea380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:48.670 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:48.687 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:48.686333 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:49.328 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:49.326142 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:49.317+0000 7f104cb46380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:49.328 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:49.317+0000 7f104cb46380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:49.328 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:49.317+0000 7f104cb46380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:49.329 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:49.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:49.377935 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:49.952 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:03:50.001 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:49.992044 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:49.985+0000 7f2ea79c5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:50.001 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:49.985+0000 7f2ea79c5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:50.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:49.985+0000 7f2ea79c5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:50.002 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:50.004 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:50.003551 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:50.570 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:50.567643 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:50.561+0000 7faf157f2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:50.571 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:50.561+0000 7faf157f2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:50.571 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:50.561+0000 7faf157f2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:50.571 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:50.618 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:03:50.618 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:03:50.435127Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:03:50.435127Z", "running": 0, "size": 1}}] 2022-03-08T23:03:50.658 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:50.657000 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:51.201 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:51.198515 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-08T23:03:51.474 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:51.472637 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:51.465+0000 7fde2601e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:51.475 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:51.465+0000 7fde2601e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:51.475 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:51.465+0000 7fde2601e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:51.475 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:51.864 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:51.862464 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:03:53.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:53.988632 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:03:53.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:53.988656 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:03:54.040 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:54.039292 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:54.638 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:54.637006 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:03:54.629+0000 7f8b68351380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:03:54.639 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:54.629+0000 7f8b68351380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:03:54.639 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:03:54.629+0000 7f8b68351380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:03:54.639 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:03:55.143 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:55.142068 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:55.149 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:55.148667 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:03:55.152 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:55.150732 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-08T23:03:55.938 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:55.936652 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:03:56.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:56.377708 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:03:56.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:56.377734 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:03:56.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:56.377740 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:03:56.952 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:56.951045 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:03:57.302 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:57.301259 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:03:57.777 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:57.776036 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:03:57.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:57.776075 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:03:57.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:57.776084 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:03:58.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:58.325442 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:03:58.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:58.325480 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:03:59.093 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:59.092301 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:03:59.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:03:59.761966 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:04:00.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:00.620959 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:04:00.613+0000 7f615181e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:04:00.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:00.613+0000 7f615181e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:04:00.624 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:00.613+0000 7f615181e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:04:00.624 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:04:00.641 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:01.394 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:01.394 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:01.180113Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:01.180113Z", "running": 0, "size": 1}}] 2022-03-08T23:04:01.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:01.981667 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:04:01.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:01.981688 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:04:05.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:05.309664 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:04:05.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:05.689108 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:04:05.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:05.689135 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:04:08.717 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.716385 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:04:08.718 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.716474 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:04:08.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.813264 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:04:08.822 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.819858 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:04:08.822 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.821001 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:04:08.822 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:08.821105 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:04:10.881 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:10.879958 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:04:11.417 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:11.549 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:11.548073 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:04:11.537+0000 7f8ea7138380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:04:11.549 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:11.537+0000 7f8ea7138380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:04:11.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:11.537+0000 7f8ea7138380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:04:11.550 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:04:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:12.107 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:11.942601Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:11.942601Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:11.942601Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:11.942601Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:11.942601Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:11.942601Z", "running": 0, "size": 1}}] 2022-03-08T23:04:12.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:12.535451 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi178": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:04:22.129 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:23.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:23.196 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:22.953557Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:22.953557Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:22.953557Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:22.953557Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:22.953557Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:22.953557Z", "running": 0, "size": 1}}] 2022-03-08T23:04:32.037 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:32.036657 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:04:32.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:04:32.661771 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:04:32.652+0000 7f9c2084d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:04:32.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:32.652+0000 7f9c2084d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:04:32.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:04:32.652+0000 7f9c2084d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:04:32.664 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:04:33.221 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:35.951 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:35.952 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:35.769363Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:35.769363Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:35.769363Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:35.769363Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:35.769363Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:35.769363Z", "running": 0, "size": 1}}] 2022-03-08T23:04:45.976 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:46.809 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:46.810 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:46.497592Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:46.497592Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:46.497592Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:46.497592Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:46.497592Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:46.497592Z", "running": 0, "size": 1}}] 2022-03-08T23:04:56.835 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:04:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:04:57.704 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:04:57.528410Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:04:57.528410Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:04:57.528410Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:04:57.528410Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:04:57.528410Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:04:57.528410Z", "running": 0, "size": 1}}] 2022-03-08T23:05:07.727 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:05:08.388 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:05:08.389 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:05:08.223055Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:05:08.223055Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:05:08.223055Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:05:08.223055Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:05:08.223055Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:05:08.223055Z", "running": 0, "size": 1}}] 2022-03-08T23:05:13.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:05:13.630805 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:05:14.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:05:14.360339 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:05:14.347+0000 7f39203a5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:05:14.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:05:14.347+0000 7f39203a5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:05:14.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:05:14.347+0000 7f39203a5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:05:14.363 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:05:18.415 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:05:19.088 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:05:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:05:18.924737Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:05:18.924737Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:05:18.924737Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:05:18.924737Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:05:18.924737Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:05:18.924737Z", "running": 0, "size": 1}}] 2022-03-08T23:05:29.116 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:05:29.825 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:05:29.826 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:05:29.645725Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:05:29.645725Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:05:29.645725Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:05:29.645725Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:05:29.645725Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:05:29.645725Z", "running": 0, "size": 1}}] 2022-03-08T23:05:36.276 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:05:36.273491 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi153": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:05:39.849 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:05:40.573 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:05:40.574 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:05:40.394957Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:05:40.394957Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:05:40.394957Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:05:40.394957Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:05:40.394957Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:05:40.394957Z", "running": 0, "size": 1}}] 2022-03-08T23:05:50.598 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:05:51.380 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:05:51.381 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:05:51.135832Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:05:51.135832Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:05:51.135832Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:05:51.135832Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:05:51.135832Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:05:51.135832Z", "running": 0, "size": 1}}] 2022-03-08T23:06:01.406 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:02.124 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:02.125 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:01.935424Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:01.935424Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:01.935424Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:01.935424Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:01.935424Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:01.935424Z", "running": 0, "size": 1}}] 2022-03-08T23:06:12.150 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:12.900 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:12.900 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:12.688604Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:12.688604Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:12.688604Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:12.688604Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:12.688604Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:12.688604Z", "running": 0, "size": 1}}] 2022-03-08T23:06:22.922 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:23.864 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:23.966 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:23.435222Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:23.435222Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:23.435222Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:23.435222Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:23.435222Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:23.435222Z", "running": 0, "size": 1}}] 2022-03-08T23:06:33.969 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:34.698 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:34.699 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:34.494912Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:34.494912Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:34.494912Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:34.494912Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:34.494912Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:34.494912Z", "running": 0, "size": 1}}] 2022-03-08T23:06:36.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:06:36.288890 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:06:36.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:06:36.994957 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:06:36.983+0000 7fc633334380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:06:36.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:06:36.983+0000 7fc633334380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:06:36.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:06:36.983+0000 7fc633334380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:06:36.999 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:06:44.719 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:45.525 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:45.525 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:45.352009Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:45.352009Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:45.352009Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:45.352009Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:45.352009Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:45.352009Z", "running": 0, "size": 1}}] 2022-03-08T23:06:55.549 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:06:56.283 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:06:56.283 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:06:56.102907Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:06:56.102907Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:06:56.102907Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:06:56.102907Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:06:56.102907Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:06:56.102907Z", "running": 0, "size": 1}}] 2022-03-08T23:07:06.309 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:07:07.056 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:07:07.056 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:07:06.861874Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:07:06.861874Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:07:06.861874Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:07:06.861874Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:07:06.861874Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:07:06.861874Z", "running": 0, "size": 1}}] 2022-03-08T23:07:17.076 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:07:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:07:17.798 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:07:17.627045Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:07:17.627045Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:07:17.627045Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:07:17.627045Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:07:17.627045Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:07:17.627045Z", "running": 0, "size": 1}}] 2022-03-08T23:07:27.825 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:07:28.543 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:07:28.543 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:07:28.373995Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:07:28.373995Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:07:28.373995Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:07:28.373995Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:07:28.373995Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:07:28.373995Z", "running": 0, "size": 1}}] 2022-03-08T23:07:38.567 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:07:39.224 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:07:39.225 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:07:39.058370Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:07:39.058370Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:07:39.058370Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:07:39.058370Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:07:39.058370Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:07:39.058370Z", "running": 0, "size": 1}}] 2022-03-08T23:07:49.246 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:07:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:07:49.981 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:07:49.797017Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:07:49.797017Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:07:49.797017Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:07:49.797017Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:07:49.797017Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:07:49.797017Z", "running": 0, "size": 1}}] 2022-03-08T23:08:00.006 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:00.670 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:00.508744Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:00.508744Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:00.508744Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:00.508744Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:00.508744Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:00.508744Z", "running": 0, "size": 1}}] 2022-03-08T23:08:10.693 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:11.445 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:11.445 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:11.222037Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:11.222037Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:11.222037Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:11.222037Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:11.222037Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:11.222037Z", "running": 0, "size": 1}}] 2022-03-08T23:08:21.468 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:22.211 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:22.212 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:22.025199Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:22.025199Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:22.025199Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:22.025199Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:22.025199Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:22.025199Z", "running": 0, "size": 1}}] 2022-03-08T23:08:32.237 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:32.923 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:32.754378Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:32.754378Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:32.754378Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:32.754378Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:32.754378Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:32.754378Z", "running": 0, "size": 1}}] 2022-03-08T23:08:42.943 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:43.724 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:43.724 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:43.543876Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:43.543876Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:43.543876Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:43.543876Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:43.543876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:43.543876Z", "running": 0, "size": 1}}] 2022-03-08T23:08:53.747 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:08:54.502 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:08:54.502 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:08:54.308179Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:08:54.308179Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:08:54.308179Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:08:54.308179Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:08:54.308179Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:08:54.308179Z", "running": 0, "size": 1}}] 2022-03-08T23:09:04.525 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:05.193 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:05.193 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:05.016629Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:05.016629Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:05.016629Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:05.016629Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:05.016629Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:05.016629Z", "running": 0, "size": 1}}] 2022-03-08T23:09:15.218 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:15.938 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:15.753292Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:15.753292Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:15.753292Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:15.753292Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:15.753292Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:15.753292Z", "running": 0, "size": 1}}] 2022-03-08T23:09:20.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:09:20.844303 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:09:21.504 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:09:21.494662 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:09:21.484+0000 7f677db38380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:09:21.504 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:09:21.484+0000 7f677db38380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:09:21.505 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:09:21.484+0000 7f677db38380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:09:21.505 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:09:25.962 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:26.768 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:26.768 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:26.585770Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:26.585770Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:26.585770Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:26.585770Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:26.585770Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:26.585770Z", "running": 0, "size": 1}}] 2022-03-08T23:09:36.791 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:37.581 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:37.581 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:37.397528Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:37.397528Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:37.397528Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:37.397528Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:37.397528Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:37.397528Z", "running": 0, "size": 1}}] 2022-03-08T23:09:47.605 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:48.345 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:48.166501Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:48.166501Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:48.166501Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:48.166501Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:48.166501Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:48.166501Z", "running": 0, "size": 1}}] 2022-03-08T23:09:58.368 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:09:59.071 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:09:59.071 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:09:58.890596Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:09:58.890596Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:09:58.890596Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:09:58.890596Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:09:58.890596Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:09:58.890596Z", "running": 0, "size": 1}}] 2022-03-08T23:10:09.092 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:10:09.761 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:10:09.761 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:10:09.576076Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:10:09.576076Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:10:09.576076Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:10:09.576076Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:10:09.576076Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:10:09.576076Z", "running": 0, "size": 1}}] 2022-03-08T23:10:19.786 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:10:20.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:10:20.476 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:10:20.294021Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:10:20.294021Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:10:20.294021Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:10:20.294021Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:10:20.294021Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:10:20.294021Z", "running": 0, "size": 1}}] 2022-03-08T23:10:30.496 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:10:31.193 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:10:31.194 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:10:31.020730Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:10:31.020730Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:10:31.020730Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:10:31.020730Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:10:31.020730Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:10:31.020730Z", "running": 0, "size": 1}}] 2022-03-08T23:10:41.217 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:10:42.025 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:10:42.025 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:10:41.767877Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:10:41.767877Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:10:41.767877Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:10:41.767877Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:10:41.767877Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:10:41.767877Z", "running": 0, "size": 1}}] 2022-03-08T23:10:52.048 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:10:52.730 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:10:52.731 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:10:52.547904Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:10:52.547904Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:10:52.547904Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:10:52.547904Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:10:52.547904Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:10:52.547904Z", "running": 0, "size": 1}}] 2022-03-08T23:11:02.756 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:03.443 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:03.444 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:03.285105Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:03.285105Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:03.285105Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:03.285105Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:03.285105Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:03.285105Z", "running": 0, "size": 1}}] 2022-03-08T23:11:13.475 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:14.178 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:14.179 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:14.004382Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:14.004382Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:14.004382Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:14.004382Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:14.004382Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:14.004382Z", "running": 0, "size": 1}}] 2022-03-08T23:11:24.202 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:25.187 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:25.188 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:24.712795Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:24.712795Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:24.712795Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:24.712795Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:24.712795Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:24.712795Z", "running": 0, "size": 1}}] 2022-03-08T23:11:35.210 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:35.919 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:35.919 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:35.751428Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:35.751428Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:35.751428Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:35.751428Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:35.751428Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:35.751428Z", "running": 0, "size": 1}}] 2022-03-08T23:11:45.942 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:46.636 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:46.636 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:46.454689Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:46.454689Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:46.454689Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:46.454689Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:46.454689Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:46.454689Z", "running": 0, "size": 1}}] 2022-03-08T23:11:56.660 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:11:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:11:57.711 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:11:57.520255Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:11:57.520255Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:11:57.520255Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:11:57.520255Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:57.520255Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:11:57.520255Z", "running": 0, "size": 1}}] 2022-03-08T23:12:07.734 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:12:08.415 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:12:08.416 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:12:08.206675Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:12:08.206675Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:12:08.206675Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:12:08.206675Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:08.206675Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:08.206675Z", "running": 0, "size": 1}}] 2022-03-08T23:12:18.441 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:12:19.231 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:12:19.231 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:12:19.003591Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:12:19.003591Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:12:19.003591Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:12:19.003591Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:19.003591Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:19.003591Z", "running": 0, "size": 1}}] 2022-03-08T23:12:29.252 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:12:29.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:12:29.924 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:12:29.746087Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:12:29.746087Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:12:29.746087Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:12:29.746087Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:29.746087Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:29.746087Z", "running": 0, "size": 1}}] 2022-03-08T23:12:39.947 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:12:40.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:12:40.660 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:12:40.470328Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:12:40.470328Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:12:40.470328Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:12:40.470328Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:40.470328Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:40.470328Z", "running": 0, "size": 1}}] 2022-03-08T23:12:50.679 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:12:51.366 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:12:51.367 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:12:51.188768Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:12:51.188768Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:12:51.188768Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:12:51.188768Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:51.188768Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:51.188768Z", "running": 0, "size": 1}}] 2022-03-08T23:13:01.387 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:02.143 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:02.143 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:01.936592Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:01.936592Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:01.936592Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:01.936592Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:01.936592Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:01.936592Z", "running": 0, "size": 1}}] 2022-03-08T23:13:12.164 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:12.886 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:12.687135Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:12.687135Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:12.687135Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:12.687135Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:12.687135Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:12.687135Z", "running": 0, "size": 1}}] 2022-03-08T23:13:22.910 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:23.640 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:23.640 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:23.449201Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:23.449201Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:23.449201Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:23.449201Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:23.449201Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:23.449201Z", "running": 0, "size": 1}}] 2022-03-08T23:13:33.666 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:34.369 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:34.370 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:34.187114Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:34.187114Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:34.187114Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:34.187114Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:34.187114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:34.187114Z", "running": 0, "size": 1}}] 2022-03-08T23:13:44.394 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:45.154 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:45.155 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:44.883744Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:44.883744Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:44.883744Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:44.883744Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:44.883744Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:44.883744Z", "running": 0, "size": 1}}] 2022-03-08T23:13:55.183 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:13:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:13:55.949 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:13:55.766670Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:13:55.766670Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:13:55.766670Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:13:55.766670Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:55.766670Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:55.766670Z", "running": 0, "size": 1}}] 2022-03-08T23:14:05.968 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:14:06.669 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:14:06.669 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:14:06.490672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:14:06.490672Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:14:06.490672Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:14:06.490672Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:06.490672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:06.490672Z", "running": 0, "size": 1}}] 2022-03-08T23:14:16.693 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:14:17.409 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:14:17.409 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:14:17.206364Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:14:17.206364Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:14:17.206364Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:14:17.206364Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:17.206364Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:17.206364Z", "running": 0, "size": 1}}] 2022-03-08T23:14:27.431 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:14:28.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:14:28.541 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:14:28.323448Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:14:28.323448Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:14:28.323448Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:14:28.323448Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:28.323448Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:28.323448Z", "running": 0, "size": 1}}] 2022-03-08T23:14:38.564 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:14:41.610 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:14:41.611 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:14:41.412608Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:14:41.412608Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:14:41.412608Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:14:41.412608Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:41.412608Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:41.412608Z", "running": 0, "size": 1}}] 2022-03-08T23:14:49.199 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:14:49.184863 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:14:49.845 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:14:49.830991 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:14:49.819+0000 7fe511554380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:14:49.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:14:49.819+0000 7fe511554380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:14:49.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:14:49.819+0000 7fe511554380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:14:49.846 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:14:51.634 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:14:52.372 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:14:52.372 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:14:52.185374Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:14:52.185374Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:14:52.185374Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:14:52.185374Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:52.185374Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:52.185374Z", "running": 0, "size": 1}}] 2022-03-08T23:15:02.394 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:03.120 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:03.120 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:02.938156Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:02.938156Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:02.938156Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:02.938156Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:02.938156Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:02.938156Z", "running": 0, "size": 1}}] 2022-03-08T23:15:13.148 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:13.892 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:13.892 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:13.684893Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:13.684893Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:13.684893Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:13.684893Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:13.684893Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:13.684893Z", "running": 0, "size": 1}}] 2022-03-08T23:15:23.941 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:24.618 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:24.618 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:24.438305Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:24.438305Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:24.438305Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:24.438305Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:24.438305Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:24.438305Z", "running": 0, "size": 1}}] 2022-03-08T23:15:34.643 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:35.357 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:35.166600Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:35.166600Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:35.166600Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:35.166600Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:35.166600Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:35.166600Z", "running": 0, "size": 1}}] 2022-03-08T23:15:45.382 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:46.090 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:46.090 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:45.897170Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:45.897170Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:45.897170Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:45.897170Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:45.897170Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:45.897170Z", "running": 0, "size": 1}}] 2022-03-08T23:15:56.115 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:15:56.808 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:15:56.808 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:15:56.631126Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:15:56.631126Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:15:56.631126Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:15:56.631126Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:56.631126Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:56.631126Z", "running": 0, "size": 1}}] 2022-03-08T23:16:06.832 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:16:07.536 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:16:07.537 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:16:07.345282Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:16:07.345282Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:16:07.345282Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:16:07.345282Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:07.345282Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:07.345282Z", "running": 0, "size": 1}}] 2022-03-08T23:16:17.558 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:16:18.242 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:16:18.242 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:16:18.072412Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:16:18.072412Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:16:18.072412Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:16:18.072412Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:18.072412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:18.072412Z", "running": 0, "size": 1}}] 2022-03-08T23:16:28.270 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:16:29.028 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:16:29.028 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:16:28.835994Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:16:28.835994Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:16:28.835994Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:16:28.835994Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:28.835994Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:28.835994Z", "running": 0, "size": 1}}] 2022-03-08T23:16:39.053 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:16:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:16:39.742 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:16:39.563287Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:16:39.563287Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:16:39.563287Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:16:39.563287Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:39.563287Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:39.563287Z", "running": 0, "size": 1}}] 2022-03-08T23:16:49.765 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:16:50.585 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:16:50.586 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:16:50.392932Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:16:50.392932Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:16:50.392932Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:16:50.392932Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:50.392932Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:50.392932Z", "running": 0, "size": 1}}] 2022-03-08T23:17:00.610 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:01.332 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:01.333 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:01.156635Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:01.156635Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:01.156635Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:01.156635Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:01.156635Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:01.156635Z", "running": 0, "size": 1}}] 2022-03-08T23:17:11.357 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:12.411 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:11.860623Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:11.860623Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:11.860623Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:11.860623Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:11.860623Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:11.860623Z", "running": 0, "size": 1}}] 2022-03-08T23:17:22.434 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:23.162 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:23.163 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:22.965901Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:22.965901Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:22.965901Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:22.965901Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:22.965901Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:22.965901Z", "running": 0, "size": 1}}] 2022-03-08T23:17:33.187 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:33.855 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:33.856 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:33.672279Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:33.672279Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:33.672279Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:33.672279Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:33.672279Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:33.672279Z", "running": 0, "size": 1}}] 2022-03-08T23:17:43.878 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:44.577 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:44.577 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:44.405511Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:44.405511Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:44.405511Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:44.405511Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:44.405511Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:44.405511Z", "running": 0, "size": 1}}] 2022-03-08T23:17:54.600 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:17:55.307 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:17:55.307 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:17:55.119012Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:17:55.119012Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:17:55.119012Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:17:55.119012Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:55.119012Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:55.119012Z", "running": 0, "size": 1}}] 2022-03-08T23:18:05.331 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:06.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:06.050 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:05.841431Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:05.841431Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:05.841431Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:05.841431Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:05.841431Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:05.841431Z", "running": 0, "size": 1}}] 2022-03-08T23:18:16.069 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:16.793 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:16.793 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:16.616538Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:16.616538Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:16.616538Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:16.616538Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:16.616538Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:16.616538Z", "running": 0, "size": 1}}] 2022-03-08T23:18:26.818 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:27.540 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:27.540 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:27.360753Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:27.360753Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:27.360753Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:27.360753Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:27.360753Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:27.360753Z", "running": 0, "size": 1}}] 2022-03-08T23:18:37.563 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:38.261 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:38.261 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:38.068206Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:38.068206Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:38.068206Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:38.068206Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:38.068206Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:38.068206Z", "running": 0, "size": 1}}] 2022-03-08T23:18:48.285 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:49.018 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:49.018 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:48.839088Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:48.839088Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:48.839088Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:48.839088Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:48.839088Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:48.839088Z", "running": 0, "size": 1}}] 2022-03-08T23:18:59.042 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:18:59.707 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:18:59.707 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:18:59.531674Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:18:59.531674Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:18:59.531674Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:18:59.531674Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:59.531674Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:59.531674Z", "running": 0, "size": 1}}] 2022-03-08T23:19:09.731 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:19:10.452 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:19:10.452 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:19:10.277170Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:19:10.277170Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:19:10.277170Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:19:10.277170Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:10.277170Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:10.277170Z", "running": 0, "size": 1}}] 2022-03-08T23:19:20.475 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:19:21.217 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:19:21.217 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:19:21.024840Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:19:21.024840Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:19:21.024840Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:19:21.024840Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:21.024840Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:21.024840Z", "running": 0, "size": 1}}] 2022-03-08T23:19:31.239 DEBUG:teuthology.orchestra.run.smithi042:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-tbfns -- ceph orch ls -f json 2022-03-08T23:19:31.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2022-03-08T23:19:31.966 INFO:teuthology.orchestra.run.smithi042.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:18.000000Z", "last_refresh": "2022-03-08T23:19:31.791070Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:58:07.000000Z", "last_refresh": "2022-03-08T23:19:31.791070Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:34.000000Z", "last_refresh": "2022-03-08T23:19:31.791070Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:59:42.000000Z", "last_refresh": "2022-03-08T23:19:31.791070Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:31.791070Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:31.791070Z", "running": 0, "size": 1}}] 2022-03-08T23:19:31.987 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:19:31.988 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:19:31.988 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:19:31.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:19:32.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:19:32.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:19:32.037 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:19:32.037 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:19:32.050 DEBUG:teuthology.orchestra.remote:smithi042:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:19:32.065 DEBUG:teuthology.orchestra.run.smithi042:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-08T23:19:32.139 INFO:teuthology.orchestra.run.smithi042.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:19:32.145 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:19:32.146 DEBUG:teuthology.orchestra.run.smithi042:> kubectl delete -f cluster.yaml 2022-03-08T23:19:32.224 INFO:teuthology.orchestra.run.smithi042.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:19:32.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.220077 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:19:32.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.220223 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:19:32.234 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.220560 I | operator: successfully started the controller-runtime manager 2022-03-08T23:19:32.235 INFO:tasks.rook.operator.smithi042.stdout:I0308 23:19:32.220626 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:19:32.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.265644 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:19:32.276 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.265677 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:19:32.276 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.265725 I | operator: setting up schemes 2022-03-08T23:19:32.282 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.269736 I | operator: setting up the controller-runtime manager 2022-03-08T23:19:32.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.474362 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:19:32.490 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477371 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:19:32.490 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477411 I | ceph-cluster-controller: successfully started 2022-03-08T23:19:32.490 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477467 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:19:32.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477494 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:19:32.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477540 I | ceph-block-pool-controller: successfully started 2022-03-08T23:19:32.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477577 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:19:32.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477620 I | ceph-object-realm-controller: successfully started 2022-03-08T23:19:32.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477639 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:19:32.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477656 I | ceph-object-zone-controller: successfully started 2022-03-08T23:19:32.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477862 I | ceph-object-controller: successfully started 2022-03-08T23:19:32.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477908 I | ceph-file-controller: successfully started 2022-03-08T23:19:32.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477952 I | ceph-nfs-controller: successfully started 2022-03-08T23:19:32.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.477990 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:19:32.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478043 I | ceph-client-controller: successfully started 2022-03-08T23:19:32.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478081 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:19:32.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478132 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:19:32.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478159 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:19:32.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478179 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:19:32.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478200 I | ceph-bucket-topic: successfully started 2022-03-08T23:19:32.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478213 I | ceph-bucket-notification: successfully started 2022-03-08T23:19:32.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478230 I | ceph-bucket-notification: successfully started 2022-03-08T23:19:32.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.478247 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:19:32.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.479620 I | operator: starting the controller-runtime manager 2022-03-08T23:19:32.595 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.584647 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:19:32.595 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.584717 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:19:32.595 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.584736 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:19:32.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.588875 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:19:32.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.589979 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:32.732 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.723291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:19:32.741 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:32.732361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:19:33.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:33.352281 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:33.339+0000 7fb9eb35d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:33.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:33.339+0000 7fb9eb35d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:33.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:33.339+0000 7fb9eb35d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:33.363 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:33.410 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:33.401234 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:34.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:34.115803 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:34.103+0000 7fc49b1e1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:34.126 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:34.103+0000 7fc49b1e1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:34.126 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:34.103+0000 7fc49b1e1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:34.126 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:34.146 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:34.136948 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:34.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:34.782852 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:34.771+0000 7f5d43235380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:34.793 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:34.771+0000 7f5d43235380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:34.793 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:34.771+0000 7f5d43235380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:34.793 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:34.819 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:34.809563 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:35.480 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:35.470023 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:35.459+0000 7fd87f0bd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:35.480 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:35.459+0000 7fd87f0bd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:35.480 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:35.459+0000 7fd87f0bd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:35.480 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:35.526 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:35.516362 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:36.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:36.207267 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:36.195+0000 7fb0f18e8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:36.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:36.195+0000 7fb0f18e8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:36.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:36.195+0000 7fb0f18e8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:36.218 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:36.304 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:36.295361 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:36.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:36.945891 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:36.935+0000 7f0bfc918380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:36.956 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:36.935+0000 7f0bfc918380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:36.956 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:36.935+0000 7f0bfc918380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:36.956 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:37.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:37.113008 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:37.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:37.814701 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:37.803+0000 7fae4715e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:37.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:37.803+0000 7fae4715e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:37.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:37.803+0000 7fae4715e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:37.825 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:38.154 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:38.144467 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:38.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:38.831460 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:38.819+0000 7f5d6d348380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:38.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:38.819+0000 7f5d6d348380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:38.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:38.819+0000 7f5d6d348380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:38.842 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:39.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:39.486167 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:40.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:40.186734 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:40.175+0000 7f6a9660f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:40.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:40.175+0000 7f6a9660f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:40.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:40.175+0000 7f6a9660f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:40.198 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:41.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:41.474983 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:42.128 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:42.118898 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:42.107+0000 7f957b6ae380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:42.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:42.107+0000 7f957b6ae380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:42.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:42.107+0000 7f957b6ae380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:42.129 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:42.799 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:42.790455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:19:42.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:42.837449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:19:44.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:44.687156 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:45.343 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:45.333981 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:45.323+0000 7f6178790380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:45.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:45.323+0000 7f6178790380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:45.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:45.323+0000 7f6178790380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:45.344 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:50.475 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:50.465934 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:19:51.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:51.119265 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:19:51.106+0000 7f11251c6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:19:51.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:51.106+0000 7f11251c6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:19:51.129 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:19:51.106+0000 7f11251c6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:19:51.130 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:19:52.939 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:52.930469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:19:52.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:19:52.988255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:01.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:01.368166 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:20:02.036 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:02.027117 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:20:02.014+0000 7f0fee2f9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:20:02.037 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:20:02.014+0000 7f0fee2f9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:20:02.037 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:20:02.014+0000 7f0fee2f9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:20:02.037 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:20:03.165 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:03.156465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:03.179 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:03.170084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:13.258 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:13.248791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:13.274 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:13.265346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:22.522 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:22.513482 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:20:23.177 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:23.168166 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:20:23.154+0000 7fbb1c1a5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:20:23.178 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:20:23.154+0000 7fbb1c1a5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:20:23.178 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:20:23.154+0000 7fbb1c1a5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:20:23.178 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:20:23.350 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:23.341294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:23.361 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:23.352964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:33.460 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:33.451386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:33.472 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:33.464131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:43.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:43.527281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:43.549 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:43.540650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:53.648 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:53.639415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:20:53.661 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:20:53.653028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:03.744 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:03.735066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:03.756 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:03.747803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:04.153 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:04.144594 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:21:04.862 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:04.853448 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:21:04.842+0000 7f0da401f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:21:04.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:21:04.842+0000 7f0da401f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:21:04.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:21:04.842+0000 7f0da401f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:21:04.863 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:21:13.818 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:13.809246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:13.831 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:13.822677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:23.925 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:23.916326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:23.939 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:23.931050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:34.056 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:34.047067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:34.072 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:34.063846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:44.144 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:44.135350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:44.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:44.148577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:54.303 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:54.295005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:21:54.323 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:21:54.308533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:04.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:04.448183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:04.498 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:04.489389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:14.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:14.641312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:14.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:14.656213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:24.777 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:24.768902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:24.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:24.782667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:26.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:26.782684 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:22:27.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:27.441047 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:22:27.432+0000 7f663a269380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:22:27.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:22:27.432+0000 7f663a269380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:22:27.451 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:22:27.432+0000 7f663a269380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:22:27.451 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:22:34.885 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:34.876151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:34.897 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:34.888417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:44.960 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:44.951796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:44.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:44.966207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:55.072 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:55.063634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:22:55.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:22:55.076121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:05.256 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:05.247371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:05.285 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:05.276119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:15.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:15.358710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:15.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:15.373671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:25.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:25.498031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:25.518 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:25.509491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:35.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:35.599503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:35.656 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:35.647750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:45.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:45.759018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:45.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:45.769234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:55.923 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:55.913378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:23:55.935 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:23:55.926221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:06.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:05.993046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:06.015 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:06.006080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:16.135 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:16.125906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:16.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:16.141275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:26.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:26.242703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:26.264 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:26.255166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:36.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:36.368015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:36.390 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:36.380417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:46.505 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:46.495823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:46.517 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:46.508101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:56.603 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:56.593125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:24:56.615 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:24:56.605185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:06.734 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:06.723784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:06.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:06.738021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:11.302 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:11.292019 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:25:11.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:11.937558 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:25:11.928+0000 7fe8ecd2b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:25:11.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:25:11.928+0000 7fe8ecd2b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:25:11.948 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:25:11.928+0000 7fe8ecd2b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:25:11.949 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:25:16.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:16.801884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:16.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:16.816756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:26.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:26.923792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:26.946 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:26.935867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:37.123 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:37.112630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:37.136 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:37.125830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:47.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:47.264895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:47.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:47.277516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:57.420 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:57.409430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:25:57.433 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:25:57.422922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:07.497 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:07.486053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:07.512 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:07.501852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:17.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:17.597516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:17.638 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:17.627931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:27.758 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:27.747617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:27.770 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:27.759359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:37.894 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:37.883480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:37.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:37.896050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:47.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:47.983215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:48.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:47.997495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:58.106 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:58.094987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:26:58.119 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:26:58.108034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:08.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:08.182042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:08.248 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:08.237219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:18.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:18.373592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:18.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:18.383014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:28.497 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:28.484896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:28.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:28.499511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:38.580 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:38.568326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:38.594 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:38.583131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:48.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:48.659445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:48.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:48.715702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:58.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:58.802516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:58.827 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:27:58.815781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:08.928 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:08.916186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:08.937 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:08.926097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:19.051 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:19.039954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:19.065 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:19.053932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:29.200 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:29.189416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:29.213 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:29.202287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:39.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:39.275754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:39.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:39.289839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:49.369 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:49.358834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:49.429 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:49.418596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:59.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:59.485140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:59.509 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:28:59.498791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:09.606 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:09.595881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:09.621 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:09.610807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:19.718 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:19.707782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:19.730 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:19.720425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:29.802 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:29.792412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:29.878 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:29.868125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:39.984 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:39.974207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:39.995 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:39.985174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:50.115 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:50.105192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:50.128 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:29:50.118458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:00.201 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:00.191370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:00.253 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:00.238909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:10.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:10.311741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:10.335 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:10.325834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:20.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:20.443413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:20.465 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:20.455266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:30.570 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:30.560754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:30.582 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:30.573176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:39.633 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:39.624155 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:30:40.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:40.267095 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:30:40.258+0000 7ff6f1995380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:30:40.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:30:40.258+0000 7ff6f1995380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:30:40.278 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:30:40.258+0000 7ff6f1995380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:30:40.278 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:30:40.733 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:40.723304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:40.745 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:40.735828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:50.881 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:50.872114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:50.895 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:30:50.885758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:01.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:01.004043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:01.026 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:01.017254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:11.138 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:11.128268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:11.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:11.141568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:21.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:21.284770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:21.308 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:21.299393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:31.403 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:31.393754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:31.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:31.406147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:41.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:41.474613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:41.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:41.486897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:51.582 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:51.572942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:51.595 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:31:51.585684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:01.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:01.681627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:01.701 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:01.691886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:11.771 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:11.762458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:11.829 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:11.820169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:21.923 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:21.914097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:21.936 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:21.927098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:32.040 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:32.031316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:32.054 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:32.045905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:42.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:42.148835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:42.170 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:42.161619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:52.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:52.275723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:52.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:32:52.291516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:02.505 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:02.497268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:02.519 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:02.511596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:12.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:12.604694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:12.624 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:12.616822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:22.694 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:22.686207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:22.709 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:22.701224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:32.780 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:32.772912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:32.794 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:32.787048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:42.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:42.858650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:42.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:42.873250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:52.986 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:52.978772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:52.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:33:52.991110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:03.071 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:03.064137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:03.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:03.078235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:13.155 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:13.148427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:13.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:13.161634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:23.271 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:23.264229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:23.283 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:23.276567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:33.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:33.386794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:33.407 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:33.400587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:43.516 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:43.509334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:43.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:43.524911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:53.622 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:53.614610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:53.634 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:34:53.627831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:03.713 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:03.705705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:03.798 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:03.791687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:13.876 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:13.869103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:13.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:13.881831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:23.974 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:23.967398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:23.989 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:23.982296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:34.063 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:34.056243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:34.075 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:34.068654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:44.244 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:44.236900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:44.256 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:44.249576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:54.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:54.358874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:54.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:35:54.371803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:04.476 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:04.469474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:04.489 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:04.482920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:14.553 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:14.546908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:14.565 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:14.559194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:24.655 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:24.648875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:24.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:24.663106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:34.756 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:34.749596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:34.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:34.761244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:44.882 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:44.875268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:44.895 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:44.888801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:55.026 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:55.019712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:55.039 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:36:55.032642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:05.152 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:05.145187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:05.165 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:05.159256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:15.242 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:15.234740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:15.263 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:15.251640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:25.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:25.372341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:25.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:25.386231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:35.497 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:35.490179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:35.513 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:35.507205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:45.588 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:45.581262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:45.635 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:45.628321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:55.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:55.690526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:55.710 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:37:55.704044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:05.785 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:05.778329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:05.800 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:05.793966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:15.924 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:15.916883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:15.935 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:15.928894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:26.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:25.994792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:26.015 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:26.008320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:36.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:36.117593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:36.136 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:36.128914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:46.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:46.233053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:46.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:46.245733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:56.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:56.319102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:56.341 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:38:56.333976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:06.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:06.445640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:06.465 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:06.458566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:16.537 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:16.530335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:16.551 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:16.543969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:26.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:26.655674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:26.675 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:26.668439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:36.754 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:36.743528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:36.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:36.757406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:46.836 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:46.828645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:46.848 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:46.841672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:56.938 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:56.930640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:56.952 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:39:56.944723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:07.059 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:07.051570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:07.070 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:07.062941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:17.188 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:17.180396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:17.200 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:17.192994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:27.281 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:27.273450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:27.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:27.285514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:37.400 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:37.392585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:37.433 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:37.425797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:47.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:47.528013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:47.553 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:47.545092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:57.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:57.656061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:57.678 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:40:57.670220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:07.765 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:07.757237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:07.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:07.771566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:17.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:17.880302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:17.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:17.895170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:28.007 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:27.998388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:28.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:28.012659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:35.644 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:35.635736 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:41:36.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:36.278164 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:41:36.266+0000 7fd0ae537380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:41:36.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:41:36.266+0000 7fd0ae537380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:41:36.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:41:36.266+0000 7fd0ae537380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:41:36.288 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:41:38.130 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:38.122152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:38.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:38.140626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:48.273 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:48.264804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:48.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:48.279257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:58.352 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:58.343701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:58.366 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:41:58.358319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:08.512 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:08.503797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:08.523 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:08.515371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:18.638 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:18.629709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:18.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:18.662826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:28.739 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:28.730780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:28.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:28.744059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:38.818 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:38.808969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:38.831 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:38.822722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:48.909 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:48.900750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:48.924 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:48.915335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:58.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:58.986883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:59.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:42:58.999284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:09.072 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:09.063349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:09.126 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:09.117488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:19.221 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:19.211606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:19.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:19.225772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:29.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:29.346619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:29.370 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:29.360696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:39.470 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:39.461429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:39.483 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:39.473945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:49.561 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:49.551780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:49.605 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:49.595649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:59.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:59.704575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:59.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:43:59.716267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:09.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:09.815538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:09.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:09.833175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:19.949 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:19.938935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:19.961 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:19.951292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:30.029 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:30.019165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:30.113 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:30.103444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:40.224 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:40.214032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:40.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:40.229223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:50.335 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:50.324747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:50.349 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:44:50.339068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:00.505 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:00.494664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:00.516 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:00.506012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:10.618 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:10.608004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:10.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:10.639543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:20.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:20.736860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:20.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:20.749205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:30.876 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:30.865549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:30.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:30.878135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:41.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:41.013572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:41.058 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:41.047224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:51.142 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:51.130720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:51.154 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:45:51.143869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:01.227 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:01.215811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:01.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:01.229209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:11.334 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:11.323121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:11.350 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:11.338221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:21.444 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:21.432835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:21.458 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:21.447840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:31.577 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:31.565669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:31.590 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:31.579054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:41.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:41.678280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:41.704 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:41.692498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:51.775 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:51.763480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:51.787 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:46:51.776279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:01.889 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:01.877532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:01.902 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:01.890899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:11.977 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:11.965263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:12.046 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:12.034908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:22.145 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:22.133335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:22.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:22.145464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:32.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:32.219752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:32.245 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:32.233687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:42.393 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:42.381016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:42.406 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:42.394762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:52.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:52.482774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:52.546 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:47:52.533544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:02.695 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:02.682750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:02.708 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:02.695723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:12.837 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:12.824421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:12.849 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:12.836761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:22.933 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:22.920986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:22.981 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:22.968815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:33.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:33.067760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:33.093 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:33.080937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:43.188 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:43.175483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:43.201 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:43.189082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:53.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:53.282411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:53.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:48:53.297487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:03.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:03.364400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:03.390 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:03.377161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:13.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:13.444145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:13.469 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:13.456806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:23.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:23.549914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:23.575 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:23.563185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:33.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:33.666440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:33.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:33.683176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:43.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:43.751161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:43.782 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:43.769773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:53.844 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:53.831299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:53.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:49:53.843107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:03.972 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:03.959390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:03.984 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:03.971503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:14.065 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:14.052348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:14.077 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:14.064792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:24.187 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:24.174679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:24.200 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:24.187173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:34.297 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:34.284103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:34.309 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:34.296742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:44.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:44.379713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:44.403 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:44.390595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:54.468 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:54.455660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:54.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:50:54.468475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:04.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:04.535640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:04.561 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:04.548538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:14.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:14.684023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:14.707 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:14.695453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:24.857 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:24.845234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:24.872 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:24.859494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:35.060 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:35.048067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:35.073 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:35.060708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:45.319 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:45.306917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:45.333 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:45.321131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:55.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:55.386119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:55.410 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:51:55.398661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:05.504 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:05.492178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:05.515 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:05.503567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:15.630 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:15.618492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:15.644 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:15.632218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:25.774 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:25.761809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:25.788 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:25.776636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:35.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:35.890545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:35.914 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:35.902478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:45.979 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:45.966605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:45.991 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:45.979313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:56.062 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:56.050173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:56.133 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:52:56.120797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:06.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:06.227047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:06.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:06.240136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:16.354 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:16.341341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:16.366 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:16.354012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:26.473 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:26.460897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:26.488 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:26.476112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:36.604 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:36.592493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:36.619 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:36.607138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:46.704 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:46.691722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:46.761 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:46.749320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:56.898 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:56.885828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:56.909 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:53:56.896948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:07.078 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:07.066110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:07.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:07.077572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:17.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:17.157745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:17.184 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:17.172064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:27.258 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:27.244606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:27.273 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:27.260760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:37.408 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:37.396452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:37.419 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:37.407416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:47.529 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:47.516856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:47.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:47.529066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:57.614 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:57.602477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:57.628 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:54:57.616216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:07.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:07.713913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:07.741 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:07.729260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:17.797 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:17.785553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:17.870 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:17.858029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:27.976 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:27.964518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:27.991 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:27.979646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:38.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:38.086598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:38.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:38.099518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:48.216 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:48.204597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:48.230 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:48.218376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:58.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:58.332423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:58.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:55:58.344934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:08.485 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:08.473721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:08.503 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:08.491226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:18.633 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:18.558166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:18.633 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:18.571261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:28.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:28.654960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:28.734 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:28.721812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:38.878 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:38.865807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:38.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:38.879792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:48.993 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:48.980564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:49.004 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:48.992491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:59.127 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:59.115509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:59.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:56:59.129513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:09.258 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:09.245492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:09.271 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:09.259627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:19.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:19.386174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:19.412 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:19.399792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:29.560 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:29.547052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:29.573 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:29.561481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:39.649 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:39.637303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:39.659 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:39.647439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:49.733 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:49.721165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:49.782 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:49.770162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:59.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:59.834799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:59.926 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:57:59.913401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:10.051 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:10.039396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:10.065 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:10.052943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:16.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:16.351036 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-08T23:58:17.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:17.082628 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:58:17.071+0000 7fe566c3b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:58:17.096 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:58:17.071+0000 7fe566c3b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:58:17.096 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08T23:58:17.071+0000 7fe566c3b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:58:17.096 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:58:20.182 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:20.170098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:20.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:20.180925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:30.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:30.264899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:30.352 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:30.340194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:40.462 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:40.450006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:40.473 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:40.460791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:50.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:50.553338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:50.579 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:58:50.567418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:00.707 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:00.695040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:00.722 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:00.710002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:10.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:10.795353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:10.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:10.804625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:21.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:21.000734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:21.028 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:21.015613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:31.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:31.144089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:31.170 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:31.158458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:41.247 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:41.234922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:41.337 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:41.324382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:51.422 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:51.409292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:51.488 INFO:tasks.rook.operator.smithi042.stdout:2022-03-08 23:59:51.476120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:01.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:01.585961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:01.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:01.598871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:11.719 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:11.706168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:11.733 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:11.720591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:21.806 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:21.793299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:21.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:21.811403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:31.936 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:31.923696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:31.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:31.941969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:42.021 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:42.008123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:42.034 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:42.021302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:52.103 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:52.089770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:52.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:00:52.128607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:02.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:02.221645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:02.247 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:02.234283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:12.376 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:12.362938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:12.393 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:12.379878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:22.537 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:22.523587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:22.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:22.537450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:32.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:32.633372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:32.660 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:32.647667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:42.773 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:42.759381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:42.783 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:42.769692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:52.900 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:52.886132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:52.913 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:01:52.900385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:02.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:02.980424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:03.007 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:02.993771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:13.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:13.097763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:13.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:13.109087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:23.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:23.190891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:23.218 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:23.204707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:33.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:33.281887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:33.309 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:33.295613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:43.388 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:43.374966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:43.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:43.416267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:53.502 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:53.488404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:53.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:02:53.500301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:03.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:03.570699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:03.634 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:03.620721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:13.745 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:13.730755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:13.757 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:13.743431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:23.869 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:23.855340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:23.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:23.865947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:33.944 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:33.930550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:33.957 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:33.943177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:44.031 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:44.016579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:44.044 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:44.030612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:54.184 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:54.170027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:54.192 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:03:54.178788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:04.266 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:04.251698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:04.278 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:04.264554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:14.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:14.450306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:14.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:14.465392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:24.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:24.526586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:24.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:24.540641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:34.613 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:34.598223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:34.627 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:34.613059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:44.736 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:44.722080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:44.752 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:44.737470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:54.856 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:54.841936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:54.873 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:04:54.859333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:04.942 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:04.927614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:04.957 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:04.942602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:15.075 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:15.060293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:15.088 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:15.073868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:25.161 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:25.145925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:25.172 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:25.158158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:35.245 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:35.230495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:35.257 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:35.242292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:45.372 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:45.356962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:45.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:45.366141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:55.465 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:55.450374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:55.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:05:55.462169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:05.574 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:05.558538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:05.587 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:05.572453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:15.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:15.661533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:15.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:15.676526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:25.811 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:25.795526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:25.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:25.811367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:35.930 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:35.915437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:35.944 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:35.929590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:46.035 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:46.019407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:46.048 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:46.033076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:56.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:56.133482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:56.159 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:06:56.145117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:06.266 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:06.252100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:06.279 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:06.264520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:16.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:16.384124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:16.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:16.395294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:26.583 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:26.568825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:26.597 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:26.582816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:36.692 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:36.677452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:36.701 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:36.687075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:46.775 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:46.760983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:46.790 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:46.776155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:56.901 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:56.887364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:56.916 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:07:56.902823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:07.007 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:06.993136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:07.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:07.006242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:17.131 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:17.117172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:17.144 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:17.130762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:27.286 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:27.272848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:27.299 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:27.283693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:37.419 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:37.405982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:37.433 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:37.419998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:47.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:47.536126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:47.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:47.548906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:57.698 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:57.684474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:57.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:08:57.697849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:07.850 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:07.836406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:07.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:07.852770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:17.995 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:17.981642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:18.012 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:17.999558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:28.138 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:28.125040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:28.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:28.137738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:38.246 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:38.232664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:38.259 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:38.246519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:48.364 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:48.350591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:48.376 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:48.363733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:58.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:58.479519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:58.506 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:09:58.493659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:08.604 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:08.590882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:08.616 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:08.603527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:18.735 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:18.721578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:18.789 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:18.776790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:28.890 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:28.877222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:28.902 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:28.889370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:39.022 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:39.009948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:39.037 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:39.024380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:49.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:49.223993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:49.248 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:49.236115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:59.406 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:59.393522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:59.419 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:10:59.406984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:09.509 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:09.497192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:09.521 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:09.509761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:19.589 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:19.576453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:19.603 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:19.590916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:29.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:29.678785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:29.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:29.691473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:39.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:39.833983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:39.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:39.843591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:49.962 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:49.950746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:49.977 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:11:49.965866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:00.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:00.110708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:00.135 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:00.123482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:10.238 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:10.226435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:10.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:10.240815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:20.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:20.342259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:20.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:20.356626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:30.501 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:30.489986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:30.513 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:30.502481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:40.594 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:40.583209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:40.606 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:40.595303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:50.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:50.702768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:50.728 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:12:50.717266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:00.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:00.805731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:00.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:00.819234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:10.925 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:10.914322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:10.937 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:10.926883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:21.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:21.000567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:21.023 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:21.012711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:31.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:31.154711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:31.178 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:31.168193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:41.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:41.263820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:41.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:41.278172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:51.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:51.353967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:51.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:13:51.368902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:01.478 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:01.467880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:01.490 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:01.479907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:11.590 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:11.580095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:11.605 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:11.594345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:21.742 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:21.732109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:21.751 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:21.740813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:31.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:31.844051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:31.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:31.856657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:41.963 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:41.952724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:41.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:41.968134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:52.086 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:52.076000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:52.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:52.087528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:57.153 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:57.143150 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T00:14:57.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:14:57.831439 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:14:57.793+0000 7f476ff55380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:14:57.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:14:57.793+0000 7f476ff55380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:14:57.843 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:14:57.793+0000 7f476ff55380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:14:57.843 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:15:02.154 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:02.144055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:02.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:02.158368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:12.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:12.252378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:12.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:12.267410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:22.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:22.375635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:22.400 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:22.389545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:32.522 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:32.512787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:32.535 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:32.525852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:42.602 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:42.592338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:42.616 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:42.606252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:52.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:52.686660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:52.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:15:52.745718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:02.868 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:02.858179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:02.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:02.871252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:12.949 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:12.939278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:12.961 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:12.951924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:23.091 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:23.081614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:23.103 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:23.093968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:33.177 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:33.167811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:33.191 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:33.181784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:43.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:43.329728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:43.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:43.344308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:53.474 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:53.464820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:53.490 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:16:53.480391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:03.571 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:03.561003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:03.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:03.575005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:13.729 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:13.719470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:13.745 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:13.735596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:23.870 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:23.861111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:23.883 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:23.874045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:33.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:33.978659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:34.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:33.990963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:44.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:44.070382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:44.091 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:44.082262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:54.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:54.164332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:54.186 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:17:54.177189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:04.306 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:04.296957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:04.322 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:04.313378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:14.400 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:14.390736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:14.412 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:14.403145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:24.483 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:24.474022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:24.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:24.482870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:34.604 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:34.594632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:34.618 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:34.608969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:44.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:44.714657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:44.737 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:44.728117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:54.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:54.802819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:54.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:18:54.816929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:04.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:04.894091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:04.917 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:04.908276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:15.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:15.015179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:15.039 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:15.029741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:25.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:25.147978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:25.171 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:25.162197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:35.249 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:35.239616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:35.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:35.252996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:45.324 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:45.315134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:45.370 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:45.361006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:55.434 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:55.424808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:55.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:19:55.438050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:05.519 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:05.510305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:05.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:05.521658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:15.665 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:15.655574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:15.684 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:15.670669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:25.800 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:25.791138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:25.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:25.805141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:35.942 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:35.932939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:35.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:35.946246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:46.187 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:46.177376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:46.248 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:46.193401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:56.397 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:56.387482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:56.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:20:56.471903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:06.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:06.553153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:06.643 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:06.633429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:16.819 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:16.809881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:16.832 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:16.823554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:27.019 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:27.010188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:27.036 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:27.027166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:37.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:37.089317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:37.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:37.102051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:47.245 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:47.235810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:47.258 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:47.249377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:57.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:57.376322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:57.397 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:21:57.388629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:07.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:07.497325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:07.520 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:07.511036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:17.611 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:17.598259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:17.621 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:17.612118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:27.687 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:27.677538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:27.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:27.690059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:37.781 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:37.771284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:37.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:37.785668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:47.857 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:47.848148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:47.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:47.862002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:57.980 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:57.970263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:57.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:22:57.984778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:08.116 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:08.106592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:08.135 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:08.121482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:18.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:18.197102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:18.220 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:18.211115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:28.304 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:28.294973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:28.316 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:28.307196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:38.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:38.474969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:38.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:38.486151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:48.621 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:48.611157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:48.633 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:48.623401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:58.719 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:58.708753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:58.731 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:23:58.721897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:08.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:08.861379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:08.883 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:08.874357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:18.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:18.945073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:18.969 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:18.959910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:29.041 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:29.031650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:29.054 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:29.044980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:39.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:39.185975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:39.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:39.197591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:49.322 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:49.312106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:49.338 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:49.328290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:59.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:59.404996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:59.478 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:24:59.468853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:09.590 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:09.580377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:09.600 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:09.590635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:19.709 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:19.698823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:19.722 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:19.712770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:29.797 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:29.787764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:29.811 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:29.801234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:39.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:39.889667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:39.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:39.955801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:50.064 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:50.054281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:50.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:25:50.070114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:00.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:00.163026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:00.186 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:00.176159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:10.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:10.283387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:10.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:10.300369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:20.419 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:20.408966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:20.459 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:20.449417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:30.576 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:30.565872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:30.589 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:30.579529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:40.700 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:40.690488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:40.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:40.704631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:50.845 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:50.834900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:50.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:26:50.844866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:00.986 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:00.976051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:01.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:00.992438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:11.117 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:11.106746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:11.130 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:11.120438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:21.268 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:21.257302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:21.280 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:21.269848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:31.400 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:31.389392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:31.414 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:31.403823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:41.523 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:41.512257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:41.542 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:41.531710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:51.675 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:51.664357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:51.685 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:27:51.675261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:01.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:01.755101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:01.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:01.768862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:11.856 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:11.845901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:11.893 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:11.882559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:21.967 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:21.955508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:21.979 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:21.969238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:32.088 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:32.077597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:32.187 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:32.089621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:42.176 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:42.165483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:42.188 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:42.178508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:52.313 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:52.303653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:52.325 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:28:52.315338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:02.401 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:02.391777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:02.465 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:02.456013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:12.579 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:12.569799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:12.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:12.589722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:22.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:22.655198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:22.676 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:22.667210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:32.872 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:32.862730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:32.884 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:32.874984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:42.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:42.965668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:42.991 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:42.980045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:53.054 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:53.045109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:53.087 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:29:53.077495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:03.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:03.135056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:03.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:03.253286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:13.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:13.377080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:13.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:13.389939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:23.483 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:23.474026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:23.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:23.487471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:33.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:33.660315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:33.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:33.674697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:43.801 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:43.792386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:43.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:43.803563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:53.878 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:53.868997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:53.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:30:53.924899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:04.083 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:04.074207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:04.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:04.086755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:14.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:14.201150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:14.221 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:14.212402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:24.335 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:24.326072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:24.349 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:24.340528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:34.421 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:34.412854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:34.434 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:34.425810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:37.849 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:37.840365 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T00:31:38.519 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:38.509922 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:31:38.499+0000 7f30fad5d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:31:38.520 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:31:38.499+0000 7f30fad5d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:31:38.520 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:31:38.499+0000 7f30fad5d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:31:38.520 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:31:44.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:44.498618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:44.570 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:44.559349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:54.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:54.668441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:54.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:31:54.681476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:04.769 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:04.760945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:04.855 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:04.846525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:14.995 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:14.986424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:15.006 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:14.997865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:25.067 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:25.058757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:25.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:25.073898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:35.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:35.195695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:35.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:35.208792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:45.313 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:45.305051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:45.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:45.318017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:55.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:55.389589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:55.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:32:55.403845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:05.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:05.475619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:05.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:05.487797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:15.570 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:15.561865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:15.581 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:15.573162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:25.682 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:25.673717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:25.693 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:25.684940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:35.798 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:35.789313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:35.809 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:35.801379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:45.887 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:45.878769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:45.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:45.891542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:56.015 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:56.006526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:56.031 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:33:56.023261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:06.128 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:06.119372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:06.144 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:06.136307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:16.301 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:16.293179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:16.314 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:16.305683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:26.403 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:26.395069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:26.417 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:26.408834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:36.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:36.482505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:36.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:36.547036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:46.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:46.659707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:46.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:46.675015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:56.772 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:56.763542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:56.783 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:34:56.775592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:06.894 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:06.886220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:06.908 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:06.900362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:17.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:16.991815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:17.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:17.005062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:27.107 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:27.099232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:27.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:27.113558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:37.229 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:37.220689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:37.241 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:37.233733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:47.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:47.312678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:47.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:47.371626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:57.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:57.472897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:57.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:35:57.488069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:07.601 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:07.593013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:07.613 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:07.605417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:17.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:17.688085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:17.708 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:17.700917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:27.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:27.817312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:27.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:27.831864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:37.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:37.947336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:37.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:37.958271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:48.036 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:48.028360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:48.051 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:48.043314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:58.126 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:58.118276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:58.139 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:36:58.131371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:08.247 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:08.238750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:08.259 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:08.251906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:18.329 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:18.321677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:18.383 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:18.375719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:28.480 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:28.472160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:28.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:28.485241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:38.554 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:38.546305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:38.598 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:38.590557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:48.726 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:48.718150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:48.739 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:48.731718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:58.848 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:58.840318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:58.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:37:58.863259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:08.980 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:08.972217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:08.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:08.986567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:19.073 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:19.065761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:19.084 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:19.076823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:29.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:29.231428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:29.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:29.244938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:39.324 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:39.316940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:39.338 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:39.331282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:49.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:49.434683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:49.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:49.446323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:59.534 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:59.526327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:59.543 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:38:59.535859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:09.615 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:09.607322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:09.631 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:09.623653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:19.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:19.719243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:19.737 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:19.730247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:29.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:29.797735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:29.820 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:29.812496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:39.915 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:39.907521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:39.930 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:39.922611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:50.066 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:50.059163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:50.079 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:39:50.071948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:00.185 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:00.177110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:00.198 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:00.190941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:10.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:10.269911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:10.288 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:10.281041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:20.396 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:20.388479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:20.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:20.401400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:30.486 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:30.479050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:30.500 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:30.492648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:40.602 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:40.595257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:40.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:40.604620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:50.715 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:50.707795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:50.728 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:40:50.721368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:00.840 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:00.832283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:00.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:00.846444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:10.944 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:10.931439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:10.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:10.947825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:21.061 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:21.053877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:21.074 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:21.066533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:31.164 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:31.156300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:31.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:31.167794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:41.270 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:41.262368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:41.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:41.276992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:51.361 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:51.353697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:51.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:41:51.415302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:01.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:01.522343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:01.542 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:01.534749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:11.641 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:11.633617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:11.658 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:11.651241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:21.758 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:21.750505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:21.773 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:21.765706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:31.833 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:31.826269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:31.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:31.839515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:41.950 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:41.943049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:41.964 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:41.957283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:52.041 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:52.033670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:52.053 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:42:52.046190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:02.121 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:02.114063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:02.134 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:02.127232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:12.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:12.229199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:12.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:12.244791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:22.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:22.349078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:22.370 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:22.362843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:32.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:32.473742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:32.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:32.488823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:42.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:42.601369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:42.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:42.615898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:52.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:52.715895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:52.739 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:43:52.731482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:02.815 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:02.807682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:02.828 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:02.821569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:12.933 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:12.926284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:12.950 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:12.940263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:23.061 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:23.053183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:23.078 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:23.071420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:33.191 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:33.183924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:33.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:33.197690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:43.279 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:43.272379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:43.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:43.284832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:53.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:53.386013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:53.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:44:53.398815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:03.476 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:03.469127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:03.489 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:03.482303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:13.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:13.554682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:13.574 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:13.567259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:23.642 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:23.634714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:23.651 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:23.644774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:33.772 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:33.765039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:33.786 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:33.779039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:43.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:43.846436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:43.867 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:43.860780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:53.968 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:53.961163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:53.980 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:45:53.972928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:04.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:04.073804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:04.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:04.088542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:14.213 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:14.206100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:14.229 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:14.222436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:24.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:24.348506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:24.369 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:24.362851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:34.440 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:34.433378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:34.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:34.445913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:44.519 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:44.512312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:44.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:44.523571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:54.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:54.601912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:54.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:46:54.616459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:04.731 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:04.724518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:04.744 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:04.737994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:14.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:14.840706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:14.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:14.853004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:24.929 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:24.922128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:24.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:24.976593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:35.116 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:35.109766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:35.130 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:35.123694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:45.244 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:45.237157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:45.254 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:45.247184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:55.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:55.355151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:55.375 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:47:55.368639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:05.496 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:05.489253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:05.509 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:05.502762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:15.620 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:15.612740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:15.654 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:15.647183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:18.525 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:18.517984 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T00:48:19.184 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:19.177464 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:48:19.170+0000 7f1cdd125380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:48:19.185 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:48:19.170+0000 7f1cdd125380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:48:19.185 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T00:48:19.170+0000 7f1cdd125380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:48:19.185 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:48:25.764 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:25.757698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:25.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:25.771748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:35.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:35.857767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:35.965 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:35.870878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:46.068 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:46.061310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:46.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:46.074852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:56.213 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:56.206671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:56.224 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:48:56.217770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:06.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:06.284852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:06.303 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:06.296874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:16.374 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:16.367791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:16.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:16.380145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:26.503 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:26.492244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:26.515 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:26.508886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:36.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:36.614607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:36.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:36.626380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:46.760 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:46.752917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:46.772 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:46.765612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:56.859 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:56.852238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:56.872 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:49:56.865732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:07.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:06.993591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:07.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:07.007282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:17.078 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:17.071569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:17.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:17.113699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:27.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:27.187117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:27.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:27.197517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:37.273 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:37.266256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:37.285 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:37.278962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:47.392 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:47.385689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:47.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:47.399002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:57.483 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:57.476579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:57.495 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:50:57.489348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:07.594 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:07.587347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:07.605 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:07.599449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:17.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:17.664816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:17.687 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:17.680472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:27.796 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:27.789415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:27.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:27.805637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:37.923 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:37.915840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:37.939 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:37.933306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:48.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:47.995517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:48.014 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:48.007633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:58.096 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:58.089488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:58.113 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:51:58.103895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:08.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:08.186806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:08.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:08.203149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:18.323 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:18.316407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:18.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:18.329011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:28.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:28.423762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:28.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:28.436562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:38.544 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:38.532781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:38.553 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:38.546929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:48.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:48.617044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:48.637 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:48.630701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:58.741 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:58.734989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:58.755 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:52:58.748660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:08.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:08.823567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:08.842 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:08.836329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:18.938 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:18.931383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:18.949 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:18.943384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:29.040 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:29.033755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:29.052 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:29.046452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:39.192 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:39.185867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:39.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:39.199922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:49.325 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:49.318484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:49.337 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:49.331418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:59.434 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:59.427491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:59.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:53:59.440321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:09.576 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:09.569327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:09.588 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:09.581646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:19.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:19.684921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:19.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:19.697487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:29.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:29.801345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:29.818 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:29.811884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:39.927 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:39.920828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:39.942 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:39.936381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:50.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:50.014103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:50.099 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:54:50.092612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:00.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:00.228457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:00.246 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:00.239857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:10.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:10.371681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:10.390 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:10.383773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:20.468 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:20.461502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:20.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:20.529358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:30.639 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:30.632762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:30.651 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:30.645058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:40.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:40.740415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:40.761 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:40.755178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:50.901 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:50.894021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:50.915 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:55:50.909100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:01.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:00.995342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:01.018 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:01.012114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:11.101 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:11.094297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:11.166 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:11.160168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:21.278 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:21.270930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:21.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:21.284845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:31.407 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:31.397790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:31.420 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:31.414000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:41.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:41.541731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:41.558 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:41.552219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:51.638 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:51.631821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:51.652 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:56:51.646306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:01.743 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:01.736536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:01.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:01.753252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:11.875 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:11.868997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:11.893 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:11.886223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:22.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:21.993044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:22.012 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:22.006240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:32.083 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:32.077086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:32.099 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:32.092502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:42.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:42.190813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:42.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:42.203748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:52.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:52.262447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:52.283 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:57:52.276846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:02.414 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:02.406867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:02.427 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:02.419979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:12.499 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:12.492501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:12.544 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:12.537985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:22.643 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:22.636877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:22.656 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:22.649348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:32.738 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:32.731572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:32.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:32.784547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:42.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:42.856393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:42.915 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:42.908931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:52.982 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:52.972639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:53.038 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:58:53.032124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:03.161 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:03.154371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:03.173 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:03.166640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:13.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:13.286075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:13.312 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:13.301391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:23.424 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:23.416840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:23.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:23.446075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:33.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:33.592592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:33.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:33.603079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:43.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:43.688505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:43.708 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:43.701716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:53.782 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:53.775429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:53.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 00:59:53.788622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:03.874 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:03.867426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:03.886 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:03.879207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:14.012 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:14.005067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:14.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:14.017389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:24.106 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:24.099753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:24.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:24.111209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:34.186 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:34.179183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:34.198 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:34.191857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:44.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:44.349669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:44.368 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:44.361506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:54.471 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:54.464181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:54.484 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:00:54.477612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:04.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:04.594353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:04.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:04.605951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:14.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:14.742982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:14.762 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:14.755628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:24.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:24.881375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:24.901 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:24.894357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:35.018 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:35.010791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:35.029 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:35.021957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:45.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:45.090042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:45.112 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:45.104542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:55.216 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:55.209065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:55.228 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:01:55.221046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:05.305 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:05.298000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:05.319 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:05.311955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:15.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:15.408040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:15.426 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:15.419819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:25.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:25.485699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:25.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:25.500717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:35.573 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:35.566159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:35.586 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:35.579794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:45.665 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:45.657906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:45.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:45.670252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:55.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:55.795671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:55.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:02:55.811027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:05.921 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:05.914419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:05.932 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:05.925026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:16.044 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:16.036712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:16.056 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:16.049075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:26.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:26.188622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:26.211 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:26.203891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:36.283 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:36.276313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:36.297 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:36.290756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:46.364 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:46.356729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:46.377 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:46.370293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:56.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:56.450136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:56.468 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:03:56.461050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:06.594 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:06.586325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:06.604 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:06.597523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:16.681 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:16.673426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:16.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:16.688963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:26.797 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:26.789882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:26.811 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:26.803955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:36.879 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:36.871369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:36.944 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:36.937465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:47.019 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:47.012243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:47.031 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:47.024742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:57.104 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:57.096507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:57.115 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:57.108168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:59.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:59.188914 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T01:04:59.838 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:04:59.830587 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:04:59.822+0000 7fd28fd9b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:04:59.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:04:59.822+0000 7fd28fd9b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:04:59.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:04:59.822+0000 7fd28fd9b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:04:59.839 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:05:07.212 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:07.204491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:07.223 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:07.215974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:17.318 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:17.310976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:17.331 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:17.324352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:27.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:27.401358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:27.467 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:27.459819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:37.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:37.558942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:37.580 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:37.573452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:47.688 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:47.680277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:47.701 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:47.693881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:57.838 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:57.830671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:57.849 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:05:57.841543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:07.917 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:07.909818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:07.932 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:07.924785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:18.005 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:17.997641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:18.049 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:18.041884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:28.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:28.117676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:28.135 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:28.128077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:38.214 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:38.206560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:38.228 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:38.221124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:48.297 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:48.289205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:48.308 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:48.301075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:58.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:58.370703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:58.389 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:06:58.381521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:08.518 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:08.510703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:08.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:08.525680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:18.644 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:18.636369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:18.660 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:18.652792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:28.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:28.783023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:28.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:28.796751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:38.876 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:38.868300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:38.890 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:38.882625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:48.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:48.989206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:49.014 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:49.006459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:59.078 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:59.070556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:59.092 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:07:59.084833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:09.201 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:09.193491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:09.216 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:09.208667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:19.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:19.276369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:19.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:19.287333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:29.407 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:29.399418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:29.420 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:29.412250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:39.515 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:39.507062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:39.529 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:39.521537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:49.633 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:49.624861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:49.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:49.639412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:59.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:59.706329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:59.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:08:59.719661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:09.800 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:09.791970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:09.857 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:09.849556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:19.935 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:19.925518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:20.010 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:20.001870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:30.081 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:30.072521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:30.093 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:30.085861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:40.212 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:40.204239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:40.226 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:40.218034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:50.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:50.291728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:50.314 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:09:50.306055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:00.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:00.377982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:00.401 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:00.393251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:10.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:10.506551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:10.528 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:10.520014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:20.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:20.624027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:20.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:20.638886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:30.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:30.714342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:30.774 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:30.766114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:40.963 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:40.954993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:40.976 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:40.968776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:51.057 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:51.048013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:51.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:10:51.103443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:01.232 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:01.222902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:01.245 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:01.237299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:11.360 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:11.351132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:11.373 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:11.364926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:21.440 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:21.431793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:21.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:21.444769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:31.606 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:31.597684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:31.622 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:31.613796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:41.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:41.738526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:41.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:41.750963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:51.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:51.890363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:51.909 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:11:51.901381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:01.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:01.989534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:02.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:02.002651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:12.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:12.113521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:12.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:12.132794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:22.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:22.208985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:22.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:22.222820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:32.311 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:32.302484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:32.324 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:32.315752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:42.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:42.421031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:42.441 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:42.432514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:52.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:52.600139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:52.619 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:12:52.611312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:02.722 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:02.713533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:02.735 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:02.726886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:12.835 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:12.826443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:12.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:12.838350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:22.970 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:22.961064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:22.983 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:22.974780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:33.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:33.076620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:33.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:33.089484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:43.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:43.194740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:43.216 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:43.207972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:53.289 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:53.280250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:53.304 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:13:53.295721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:03.379 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:03.370640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:03.391 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:03.382421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:13.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:13.522194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:13.540 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:13.531339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:23.672 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:23.663636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:23.686 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:23.677256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:33.761 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:33.752131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:33.774 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:33.765572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:43.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:43.843447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:43.865 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:43.856073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:53.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:53.925523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:54.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:14:54.015367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:04.100 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:04.091172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:04.112 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:04.103813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:14.186 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:14.176910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:14.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:14.230269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:24.341 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:24.331629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:24.354 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:24.345572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:34.453 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:34.441572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:34.462 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:34.452933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:44.582 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:44.572787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:44.593 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:44.583798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:54.657 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:54.647846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:54.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:15:54.661537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:04.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:04.737967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:04.758 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:04.748984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:14.875 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:14.865813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:14.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:14.879237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:24.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:24.989143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:25.012 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:25.003254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:35.172 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:35.162878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:35.187 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:35.177758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:45.260 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:45.250934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:45.272 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:45.262646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:55.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:55.346494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:55.369 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:16:55.360170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:05.437 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:05.427431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:05.500 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:05.490909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:15.565 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:15.555530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:15.578 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:15.568770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:25.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:25.669948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:25.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:25.681276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:35.796 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:35.786898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:35.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:35.802981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:45.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:45.878277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:45.902 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:45.892165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:55.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:55.988769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:56.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:17:56.001785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:06.103 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:06.092810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:06.117 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:06.107566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:16.192 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:16.182569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:16.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:16.196365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:26.279 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:26.269324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:26.289 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:26.280145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:36.413 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:36.403824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:36.425 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:36.415822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:46.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:46.501885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:46.551 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:46.541679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:56.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:56.658546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:56.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:18:56.669731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:06.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:06.792885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:06.815 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:06.805415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:16.889 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:16.879947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:16.903 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:16.893318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:26.998 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:26.988845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:27.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:27.001905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:37.087 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:37.077433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:37.137 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:37.127506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:47.296 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:47.286685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:47.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:47.300445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:57.417 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:57.407150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:57.430 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:19:57.420614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:07.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:07.484502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:07.508 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:07.498633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:17.586 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:17.577044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:17.599 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:17.589878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:27.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:27.673084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:27.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:27.740805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:37.858 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:37.848514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:37.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:37.861595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:47.974 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:47.964739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:47.987 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:47.978528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:58.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:58.110744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:58.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:20:58.123460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:08.200 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:08.190649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:08.209 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:08.200414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:18.312 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:18.302866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:18.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:18.317304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:28.522 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:28.512539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:28.606 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:28.597190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:38.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:38.673616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:38.739 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:38.729759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:39.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:39.842564 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T01:21:40.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:40.482353 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:21:40.472+0000 7efc0ae6d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:21:40.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:21:40.472+0000 7efc0ae6d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:21:40.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:21:40.472+0000 7efc0ae6d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:21:40.493 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:21:48.892 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:48.883346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:48.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:48.898513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:59.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:58.999415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:59.021 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:21:59.012377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:09.134 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:09.125098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:09.149 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:09.140243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:19.221 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:19.211788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:19.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:19.226079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:29.338 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:29.328844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:29.350 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:29.341616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:39.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:39.438294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:39.458 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:39.448946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:49.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:49.546336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:49.568 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:49.559279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:59.646 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:59.636830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:59.658 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:22:59.649220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:09.784 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:09.775643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:09.795 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:09.786907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:19.942 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:19.932968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:19.954 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:19.945603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:30.038 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:30.028242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:30.049 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:30.040589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:40.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:40.113513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:40.134 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:40.125666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:50.254 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:50.245500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:50.270 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:23:50.261055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:00.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:00.388866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:00.414 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:00.405561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:10.494 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:10.485095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:10.537 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:10.528362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:20.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:20.655269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:20.676 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:20.667530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:30.739 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:30.728422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:30.757 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:30.744923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:40.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:40.843093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:40.864 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:40.855262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:50.930 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:50.921418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:50.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:24:50.981881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:01.079 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:01.070578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:01.093 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:01.085062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:11.218 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:11.209177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:11.228 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:11.220033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:21.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:21.317112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:21.340 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:21.331464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:31.403 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:31.395180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:31.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:31.406860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:41.537 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:41.528952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:41.552 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:41.543590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:51.659 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:51.650160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:51.671 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:25:51.662863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:01.732 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:01.724118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:01.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:01.739167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:11.821 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:11.812665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:11.834 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:11.826415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:21.938 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:21.929950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:21.957 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:21.949167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:32.070 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:32.061458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:32.086 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:32.078344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:42.196 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:42.187931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:42.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:42.198731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:52.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:52.317509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:52.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:26:52.331275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:02.442 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:02.433839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:02.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:02.448308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:12.577 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:12.568276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:12.590 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:12.582425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:22.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:22.694892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:22.715 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:22.706994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:32.793 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:32.783879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:32.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:32.795332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:42.882 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:42.873483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:42.894 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:42.886186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:52.961 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:52.952298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:52.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:27:52.966942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:03.083 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:03.074571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:03.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:03.086701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:13.203 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:13.194222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:13.215 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:13.207604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:23.277 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:23.269390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:23.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:23.283158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:33.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:33.344658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:33.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:33.357605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:43.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:43.434736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:43.455 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:43.447183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:53.532 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:53.524102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:53.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:28:53.539791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:03.622 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:03.614261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:03.635 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:03.627726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:13.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:13.706035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:13.781 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:13.773277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:23.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:23.846203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:23.871 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:23.863360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:33.984 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:33.976164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:33.997 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:33.989603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:44.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:44.081223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:44.102 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:44.094503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:54.248 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:54.239721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:54.260 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:29:54.251750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:04.320 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:04.308985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:04.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:04.357377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:14.507 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:14.499227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:14.519 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:14.510864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:24.595 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:24.586730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:24.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:24.600918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:34.725 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:34.716815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:34.738 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:34.730166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:44.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:44.809288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:44.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:44.822532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:54.953 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:54.945560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:54.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:30:54.958330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:05.071 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:05.063385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:05.083 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:05.075323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:15.180 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:15.171863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:15.192 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:15.184166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:25.286 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:25.278479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:25.301 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:25.293006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:35.372 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:35.363634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:35.422 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:35.414447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:45.525 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:45.517237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:45.538 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:45.529976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:55.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:55.675327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:55.693 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:31:55.685885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:05.802 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:05.794367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:05.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:05.806285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:15.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:15.897729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:15.918 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:15.910777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:26.039 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:26.031191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:26.053 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:26.045080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:36.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:36.160481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:36.183 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:36.175752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:46.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:46.276718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:46.296 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:46.288735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:56.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:56.359744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:56.380 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:32:56.372905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:06.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:06.440627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:06.461 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:06.453379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:16.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:16.523092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:16.589 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:16.581353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:26.657 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:26.648877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:26.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:26.662561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:36.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:36.741286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:36.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:36.751262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:46.867 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:46.858900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:46.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:46.872923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:56.973 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:56.965188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:56.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:33:56.980169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:07.052 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:07.044518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:07.073 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:07.065103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:17.146 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:17.138485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:17.214 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:17.206736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:27.278 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:27.270464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:27.290 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:27.282492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:37.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:37.391600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:37.427 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:37.419410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:47.539 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:47.531631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:47.553 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:47.545326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:57.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:57.655975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:57.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:34:57.668929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:07.746 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:07.738026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:07.805 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:07.796642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:17.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:17.872604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:17.945 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:17.937235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:28.025 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:28.013561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:28.033 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:28.025539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:38.133 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:38.125797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:38.144 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:38.136521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:48.257 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:48.248942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:48.270 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:48.262694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:58.342 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:58.334727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:58.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:35:58.348199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:08.526 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:08.517886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:08.539 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:08.531992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:18.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:18.706325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:18.727 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:18.719521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:28.804 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:28.796640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:28.814 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:28.806736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:38.881 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:38.873687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:38.893 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:38.885780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:49.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:49.090123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:49.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:49.103212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:59.187 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:59.179732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:59.202 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:36:59.194687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:09.298 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:09.289803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:09.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:09.302689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:19.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:19.374465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:19.441 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:19.431301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:29.508 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:29.500606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:29.563 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:29.555335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:39.687 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:39.678919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:39.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:39.689787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:49.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:49.771475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:49.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:49.785038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:59.900 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:59.892278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:59.914 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:37:59.905986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:10.030 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:10.022311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:10.043 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:10.035302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:20.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:20.142568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:20.163 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:20.155665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:20.497 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:20.489686 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T01:38:21.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:21.160508 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:38:21.150+0000 7f6d0e884380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:38:21.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:38:21.154+0000 7f6d0e884380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:38:21.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:38:21.154+0000 7f6d0e884380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:38:21.170 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:38:30.281 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:30.272844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:30.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:30.286193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:40.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:40.357302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:40.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:40.370728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:50.456 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:50.448254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:50.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:38:50.523627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:00.603 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:00.594915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:00.616 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:00.608757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:10.689 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:10.681275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:10.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:10.694808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:20.801 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:20.793505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:20.813 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:20.805306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:30.930 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:30.922441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:30.943 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:30.935235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:41.019 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:41.011467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:41.033 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:41.025329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:51.099 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:51.091111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:51.141 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:39:51.132862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:01.215 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:01.207163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:01.227 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:01.219164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:11.342 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:11.334433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:11.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:11.348572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:21.429 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:21.420961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:21.442 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:21.433908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:31.523 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:31.514582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:31.535 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:31.527188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:41.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:41.600744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:41.624 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:41.616046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:51.705 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:51.696644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:51.717 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:40:51.709618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:01.815 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:01.806758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:01.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:01.818618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:11.924 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:11.915783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:11.937 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:11.929165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:22.006 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:21.997719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:22.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:22.012229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:32.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:32.148365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:32.168 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:32.160443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:42.276 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:42.268024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:42.290 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:42.281892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:52.385 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:52.376858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:52.397 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:41:52.389573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:02.473 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:02.465355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:02.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:02.524767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:12.637 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:12.628896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:12.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:12.642721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:22.763 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:22.754316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:22.776 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:22.767771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:32.876 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:32.867669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:32.890 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:32.881682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:42.999 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:42.990625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:43.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:43.002970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:53.115 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:53.106789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:53.130 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:42:53.122205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:03.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:03.228290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:03.246 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:03.238682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:13.313 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:13.304690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:13.326 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:13.317708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:23.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:23.400544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:23.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:23.414822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:33.559 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:33.550659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:33.572 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:33.563872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:43.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:43.623535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:43.646 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:43.636908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:53.713 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:53.704502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:53.724 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:43:53.716529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:03.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:03.783004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:03.873 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:03.864521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:13.984 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:13.976076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:13.996 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:13.988648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:24.059 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:24.051130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:24.109 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:24.101366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:34.189 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:34.180813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:34.202 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:34.194176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:44.280 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:44.272315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:44.290 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:44.281731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:54.363 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:54.355146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:54.375 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:44:54.366689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:04.441 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:04.432982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:04.454 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:04.446378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:14.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:14.521942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:14.544 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:14.536122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:24.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:24.603601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:24.629 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:24.621168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:34.743 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:34.734546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:34.756 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:34.748203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:44.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:44.830724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:44.851 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:44.843014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:54.954 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:54.945263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:54.969 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:45:54.961223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:05.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:05.080763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:05.102 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:05.094200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:15.177 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:15.168810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:15.191 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:15.182823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:25.297 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:25.288819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:25.310 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:25.301658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:35.411 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:35.402326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:35.425 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:35.416597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:45.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:45.524954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:45.545 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:45.537134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:55.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:55.616602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:55.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:46:55.688365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:05.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:05.808610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:05.830 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:05.822054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:15.937 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:15.928574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:15.951 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:15.942304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:26.068 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:26.059896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:26.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:26.071672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:36.222 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:36.212944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:36.231 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:36.223346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:46.345 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:46.336487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:46.357 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:46.348501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:56.466 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:56.457391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:56.479 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:47:56.470279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:06.616 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:06.607694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:06.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:06.620822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:16.716 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:16.707583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:16.729 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:16.720145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:26.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:26.854478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:26.875 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:26.866211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:36.990 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:36.981229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:37.002 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:36.993769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:47.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:47.076897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:47.143 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:47.134494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:57.237 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:57.228192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:57.249 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:48:57.240478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:07.356 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:07.347349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:07.369 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:07.360858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:17.434 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:17.425982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:17.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:17.502715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:27.579 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:27.570712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:27.592 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:27.583861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:37.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:37.688602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:37.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:37.702312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:47.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:47.770591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:47.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:47.782379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:57.849 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:57.840237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:57.864 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:49:57.855399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:07.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:07.925087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:07.947 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:07.938578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:18.026 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:18.017813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:18.061 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:18.052503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:28.160 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:28.151562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:28.172 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:28.163721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:38.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:38.231964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:38.253 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:38.244660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:48.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:48.385266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:48.422 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:48.413101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:58.527 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:58.518649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:58.544 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:50:58.535910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:08.626 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:08.617343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:08.678 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:08.669731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:18.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:18.771018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:18.790 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:18.781527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:28.856 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:28.846738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:28.911 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:28.902235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:39.053 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:39.044462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:39.069 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:39.060771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:49.200 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:49.191316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:49.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:49.201644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:59.319 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:59.310565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:59.359 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:51:59.350107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:09.513 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:09.505010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:09.533 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:09.525369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:19.642 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:19.633460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:19.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:19.662003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:29.740 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:29.731593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:29.753 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:29.744680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:39.851 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:39.842732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:39.883 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:39.874861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:50.004 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:49.996120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:50.018 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:52:50.010164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:00.079 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:00.069464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:00.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:00.080743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:10.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:10.147930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:10.171 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:10.162613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:20.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:20.231857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:20.253 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:20.244719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:30.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:30.312838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:30.333 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:30.324911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:40.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:40.438205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:40.459 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:40.450668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:50.588 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:50.578904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:50.600 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:53:50.591891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:00.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:00.661072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:00.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:00.673916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:10.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:10.742225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:10.760 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:10.752389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:20.893 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:20.884317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:20.908 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:20.899554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:30.980 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:30.971389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:30.994 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:30.985533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:41.100 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:41.091707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:41.113 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:41.104641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:51.191 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:51.182195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:51.203 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:54:51.195445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:01.180 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:01.171351 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T01:55:01.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:01.260695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:01.281 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:01.273143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:01.862 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:01.853465 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:55:01.846+0000 7f193fb31380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:55:01.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:55:01.846+0000 7f193fb31380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:55:01.863 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T01:55:01.846+0000 7f193fb31380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:55:01.863 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:55:11.357 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:11.348367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:11.370 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:11.362083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:21.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:21.455538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:21.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:21.468762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:31.555 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:31.546931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:31.587 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:31.571572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:41.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:41.637294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:41.659 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:41.650788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:51.731 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:51.722789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:51.744 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:55:51.735433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:01.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:01.851735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:01.873 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:01.864477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:11.939 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:11.931055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:11.989 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:11.981438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:22.111 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:22.101596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:22.125 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:22.116805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:32.227 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:32.218367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:32.237 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:32.229414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:42.304 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:42.295991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:42.318 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:42.310102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:52.448 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:52.439590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:52.461 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:56:52.452238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:02.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:02.528017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:02.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:02.541951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:12.650 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:12.641692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:12.662 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:12.653460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:22.731 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:22.722563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:22.747 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:22.738721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:32.832 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:32.824111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:32.845 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:32.837482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:43.009 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:43.001026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:43.019 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:43.011733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:53.116 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:53.107437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:53.128 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:57:53.120251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:03.190 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:03.181752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:03.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:03.195962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:13.279 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:13.270363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:13.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:13.285701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:23.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:23.359230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:23.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:23.374374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:33.497 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:33.488579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:33.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:33.502899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:43.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:43.639423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:43.661 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:43.652974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:53.735 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:53.726778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:53.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:58:53.740565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:03.817 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:03.808597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:03.892 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:03.884211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:13.958 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:13.949655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:14.009 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:14.001597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:24.079 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:24.070463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:24.088 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:24.079921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:34.221 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:34.212848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:34.233 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:34.224843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:44.376 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:44.367884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:44.389 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:44.381077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:54.511 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:54.502791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:54.524 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 01:59:54.516919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:04.620 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:04.611478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:04.631 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:04.623259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:14.704 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:14.696340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:14.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:14.757987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:24.848 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:24.839416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:24.861 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:24.853823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:34.976 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:34.967847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:34.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:34.980860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:45.078 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:45.069970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:45.114 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:45.083941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:55.176 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:55.167828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:55.190 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:00:55.182559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:05.320 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:05.311801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:05.333 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:05.325436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:15.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:15.441499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:15.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:15.454963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:25.567 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:25.559261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:25.580 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:25.572396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:35.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:35.674786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:35.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:35.691180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:45.775 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:45.767269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:45.833 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:45.824989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:55.898 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:55.890184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:55.913 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:01:55.902768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:06.000 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:05.992391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:06.012 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:06.004620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:16.091 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:16.082491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:16.105 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:16.097055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:26.216 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:26.207697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:26.262 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:26.254725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:36.348 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:36.340816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:36.358 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:36.350004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:46.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:46.439184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:46.459 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:46.451943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:56.524 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:56.516223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:56.536 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:02:56.528847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:06.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:06.601386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:06.691 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:06.683484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:16.827 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:16.819035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:16.839 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:16.832222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:26.956 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:26.948203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:27.003 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:26.995104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:37.090 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:37.082326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:37.103 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:37.095819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:47.255 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:47.246585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:47.270 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:47.262485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:57.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:57.390859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:57.412 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:03:57.404794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:07.504 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:07.496858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:07.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:07.506363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:17.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:17.576272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:17.597 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:17.589238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:27.694 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:27.686545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:27.706 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:27.698438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:37.799 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:37.791339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:37.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:37.804448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:47.892 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:47.884150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:47.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:47.898001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:58.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:58.011618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:58.041 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:04:58.033114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:08.146 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:08.132185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:08.153 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:08.145364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:18.239 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:18.230637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:18.250 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:18.242998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:28.375 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:28.367259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:28.388 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:28.380481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:38.462 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:38.454721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:38.475 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:38.468054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:48.568 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:48.560059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:48.582 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:48.574799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:58.697 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:58.689543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:58.710 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:05:58.702144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:08.815 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:08.807983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:08.825 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:08.817319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:18.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:18.911882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:18.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:18.926766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:29.009 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:29.001026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:29.022 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:29.014824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:39.133 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:39.125335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:39.146 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:39.138669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:49.297 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:49.289391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:49.309 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:49.301834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:59.404 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:59.396869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:59.416 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:06:59.408337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:09.534 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:09.526136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:09.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:09.540509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:19.670 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:19.662539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:19.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:19.675896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:29.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:29.770659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:29.791 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:29.784000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:39.854 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:39.846598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:39.922 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:39.914739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:50.025 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:50.017649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:50.039 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:07:50.031196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:00.135 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:00.127721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:00.150 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:00.138955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:10.219 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:10.211565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:10.275 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:10.224189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:20.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:20.318902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:20.342 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:20.334630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:30.423 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:30.415067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:30.437 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:30.428159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:40.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:40.540675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:40.561 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:40.553209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:50.663 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:50.655694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:50.677 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:08:50.669604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:00.824 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:00.816049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:00.834 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:00.827181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:10.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:10.897588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:10.920 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:10.912999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:20.995 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:20.987746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:21.009 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:21.001632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:31.110 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:31.102553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:31.122 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:31.114761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:41.229 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:41.221055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:41.246 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:41.238907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:51.325 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:51.317332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:51.338 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:09:51.331032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:01.450 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:01.442350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:01.464 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:01.456533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:11.563 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:11.555396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:11.578 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:11.570233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:21.694 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:21.686439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:21.707 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:21.698265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:31.780 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:31.772598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:31.792 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:31.784532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:41.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:41.858713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:41.880 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:41.872635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:51.975 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:51.966836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:51.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:10:51.980574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:02.061 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:02.053295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:02.074 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:02.066916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:12.198 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:12.189848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:12.208 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:12.200835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:22.325 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:22.317815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:22.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:22.332249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:32.406 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:32.398358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:32.417 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:32.408492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:41.867 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:41.858789 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T02:11:42.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:42.505853 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:11:42.495+0000 7fa7444c0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:11:42.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:11:42.495+0000 7fa7444c0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:11:42.515 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:11:42.495+0000 7fa7444c0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:11:42.515 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:11:42.563 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:42.555768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:42.578 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:42.570698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:52.730 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:52.722728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:52.743 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:11:52.735967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:02.834 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:02.826348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:02.891 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:02.883546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:12.991 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:12.983460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:13.005 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:12.997455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:23.117 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:23.109321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:23.132 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:23.124561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:33.289 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:33.281347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:33.301 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:33.293802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:43.395 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:43.387311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:43.412 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:43.404529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:53.481 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:53.473377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:53.493 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:12:53.485959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:03.554 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:03.546725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:03.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:03.558612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:13.666 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:13.658078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:13.679 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:13.671586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:23.811 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:23.803138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:23.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:23.815258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:33.988 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:33.980322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:34.003 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:33.995180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:44.103 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:44.095551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:44.117 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:44.109246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:54.199 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:54.191158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:54.256 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:13:54.248674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:04.315 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:04.307348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:04.376 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:04.368769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:14.479 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:14.471267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:14.491 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:14.484157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:24.559 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:24.545740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:24.609 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:24.601884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:34.680 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:34.672917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:34.695 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:34.687321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:44.797 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:44.789511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:44.812 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:44.804580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:54.909 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:54.902001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:54.925 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:14:54.918120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:05.029 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:05.021202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:05.041 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:05.033692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:15.114 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:15.105854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:15.128 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:15.120251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:25.202 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:25.194672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:25.283 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:25.275641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:35.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:35.401111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:35.422 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:35.414513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:45.521 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:45.513407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:45.535 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:45.528141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:55.607 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:55.599798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:55.620 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:15:55.612999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:05.737 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:05.729607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:05.750 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:05.742761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:15.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:15.844361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:15.888 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:15.881044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:25.976 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:25.968733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:25.987 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:25.979611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:36.075 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:36.067565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:36.087 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:36.079952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:46.145 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:46.137498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:46.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:46.148608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:56.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:56.285294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:56.304 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:16:56.296296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:06.502 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:06.493941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:06.515 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:06.507311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:16.744 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:16.736408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:16.758 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:16.750990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:26.877 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:26.869865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:26.892 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:26.884634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:36.964 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:36.956794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:36.978 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:36.970559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:47.109 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:47.101083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:47.120 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:47.113246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:57.244 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:57.236623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:57.258 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:17:57.250561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:07.403 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:07.395716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:07.415 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:07.407939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:17.492 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:17.484543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:17.551 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:17.543651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:27.685 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:27.677306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:27.698 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:27.690628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:37.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:37.770264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:37.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:37.818585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:47.955 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:47.946900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:47.967 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:47.959098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:58.109 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:58.101587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:58.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:18:58.111197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:08.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:08.189647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:08.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:08.203368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:18.286 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:18.277740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:18.298 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:18.291157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:28.386 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:28.378376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:28.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:28.390874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:38.500 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:38.492125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:38.512 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:38.504512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:48.575 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:48.567565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:48.589 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:48.582218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:58.699 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:58.691438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:58.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:19:58.704206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:08.829 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:08.820988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:08.843 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:08.834733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:18.946 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:18.938127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:18.960 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:18.952309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:29.069 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:29.061224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:29.082 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:29.074613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:39.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:39.167165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:39.189 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:39.182121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:49.279 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:49.271039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:49.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:49.286061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:59.382 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:59.374515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:59.394 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:20:59.386593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:09.513 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:09.506001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:09.526 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:09.519054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:19.649 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:19.641305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:19.661 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:19.654062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:29.768 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:29.760262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:29.780 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:29.772543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:39.870 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:39.862164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:39.950 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:39.942363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:50.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:50.080775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:50.101 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:21:50.092969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:00.185 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:00.177449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:00.198 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:00.190257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:10.352 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:10.343991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:10.365 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:10.357933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:20.501 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:20.493101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:20.584 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:20.576398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:30.720 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:30.712075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:30.732 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:30.724454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:40.849 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:40.838067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:40.965 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:40.853155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:50.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:50.958445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:50.979 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:22:50.971723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:01.067 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:01.059494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:01.080 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:01.072642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:11.154 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:11.146114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:11.205 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:11.197399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:21.280 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:21.272660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:21.294 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:21.286926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:31.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:31.347225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:31.368 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:31.360421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:41.432 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:41.423687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:41.477 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:41.469146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:51.593 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:51.585073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:51.605 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:23:51.598027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:01.683 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:01.675069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:01.696 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:01.688001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:11.774 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:11.765864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:11.785 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:11.777216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:21.865 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:21.857411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:21.877 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:21.869722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:31.971 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:31.963072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:31.984 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:31.975998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:42.085 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:42.077334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:42.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:42.089733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:52.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:52.185579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:52.208 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:24:52.198616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:02.308 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:02.300624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:02.320 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:02.312842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:12.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:12.391695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:12.456 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:12.447874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:22.569 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:22.561397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:22.582 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:22.574737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:32.702 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:32.694031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:32.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:32.703869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:42.845 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:42.837033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:42.857 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:42.849668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:52.924 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:52.915807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:52.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:25:52.926750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:03.008 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:03.000474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:03.034 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:03.020964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:13.096 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:13.088378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:13.110 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:13.102661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:23.210 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:23.202196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:23.223 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:23.215686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:33.330 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:33.322059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:33.343 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:33.335418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:43.452 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:43.443933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:43.467 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:43.459245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:53.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:53.540264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:53.560 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:26:53.552300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:03.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:03.617207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:03.682 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:03.674218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:13.784 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:13.776065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:13.799 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:13.791135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:23.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:23.897607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:23.917 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:23.909725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:34.028 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:34.020339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:34.039 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:34.031176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:44.154 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:44.146058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:44.171 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:44.162929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:54.240 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:54.232024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:54.253 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:27:54.245562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:04.321 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:04.313612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:04.334 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:04.326339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:14.454 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:14.445601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:14.467 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:14.458923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:22.521 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:22.513482 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T02:28:23.182 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:23.173671 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:28:23.163+0000 7f7cef780380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:28:23.182 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:28:23.163+0000 7f7cef780380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:28:23.183 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:28:23.163+0000 7f7cef780380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:28:23.183 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:28:24.573 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:24.565532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:24.588 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:24.580227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:34.711 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:34.702238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:34.722 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:34.714531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:44.835 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:44.826700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:44.847 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:44.839188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:55.013 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:55.004948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:55.027 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:28:55.019672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:05.095 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:05.087596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:05.109 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:05.101488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:15.260 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:15.251836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:15.272 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:15.264012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:25.344 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:25.335667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:25.357 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:25.348859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:35.436 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:35.427949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:35.452 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:35.444266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:45.538 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:45.529914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:45.553 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:45.545229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:55.655 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:55.647149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:55.667 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:29:55.659342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:05.771 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:05.762849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:05.783 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:05.774748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:15.893 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:15.884911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:15.908 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:15.900260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:26.042 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:26.033624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:26.054 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:26.046171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:36.118 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:36.110353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:36.133 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:36.124927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:46.235 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:46.227146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:46.249 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:46.240728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:56.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:56.318951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:56.340 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:30:56.332222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:06.417 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:06.408680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:06.433 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:06.425382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:16.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:16.599472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:16.620 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:16.612493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:26.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:26.681697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:26.787 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:26.778336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:36.899 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:36.890442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:36.907 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:36.898781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:47.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:47.002221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:47.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:47.016095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:57.136 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:57.127661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:57.157 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:31:57.149136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:07.238 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:07.229777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:07.252 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:07.243706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:17.355 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:17.346472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:17.368 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:17.360221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:27.471 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:27.461893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:27.483 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:27.474729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:37.608 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:37.599443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:37.623 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:37.614992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:47.704 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:47.695733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:47.723 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:47.714791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:57.823 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:57.814253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:57.836 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:32:57.827519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:07.941 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:07.932879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:07.953 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:07.944660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:18.047 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:18.038368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:18.062 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:18.053414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:28.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:28.147444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:28.172 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:28.164093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:38.246 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:38.237654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:38.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:38.286960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:48.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:48.358329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:48.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:48.372869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:58.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:58.439029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:58.506 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:33:58.497361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:08.575 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:08.566430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:08.588 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:08.580242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:18.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:18.681827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:18.703 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:18.694869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:28.808 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:28.799633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:28.819 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:28.811109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:38.962 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:38.953565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:38.978 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:38.964885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:49.056 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:49.047552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:49.067 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:49.059237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:59.184 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:59.175617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:59.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:34:59.188688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:09.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:09.344132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:09.371 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:09.362689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:19.441 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:19.432558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:19.454 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:19.446161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:29.550 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:29.541984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:29.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:29.557469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:39.678 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:39.669501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:39.690 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:39.682310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:49.832 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:49.823363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:49.841 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:49.832756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:59.958 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:59.949995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:59.971 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:35:59.962685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:10.032 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:10.023878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:10.045 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:10.037530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:20.144 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:20.135457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:20.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:20.147749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:30.269 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:30.260783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:30.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:30.276216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:40.387 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:40.378672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:40.399 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:40.391168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:50.514 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:50.506226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:50.530 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:36:50.522374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:00.642 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:00.634040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:00.655 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:00.646582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:10.729 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:10.721127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:10.749 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:10.734631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:20.822 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:20.814117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:20.837 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:20.829273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:30.940 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:30.932378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:30.951 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:30.943100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:41.059 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:41.051198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:41.071 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:41.063209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:51.204 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:51.195878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:51.219 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:37:51.210718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:01.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:01.331007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:01.353 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:01.345126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:11.443 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:11.435137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:11.457 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:11.449093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:21.528 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:21.519662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:21.541 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:21.533000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:31.649 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:31.640956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:31.661 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:31.653562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:41.745 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:41.736799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:41.757 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:41.749490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:51.835 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:51.827128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:51.845 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:38:51.837629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:01.906 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:01.898142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:01.966 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:01.958060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:12.077 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:12.068903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:12.089 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:12.081869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:22.156 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:22.148544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:22.170 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:22.162148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:32.268 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:32.259779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:32.281 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:32.273255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:42.378 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:42.370292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:42.393 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:42.384935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:52.463 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:52.455392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:52.475 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:39:52.467338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:02.570 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:02.562324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:02.586 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:02.577871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:12.664 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:12.655765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:12.714 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:12.706599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:22.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:22.818134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:22.840 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:22.832115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:32.921 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:32.913397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:32.932 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:32.924644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:43.049 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:43.041353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:43.062 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:43.054755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:53.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:53.155441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:53.175 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:40:53.167759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:03.292 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:03.283970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:03.362 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:03.297017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:13.413 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:13.404949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:13.426 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:13.418271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:23.546 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:23.538417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:23.560 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:23.552527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:33.658 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:33.649874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:33.669 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:33.661127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:43.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:43.758060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:43.779 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:43.771471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:53.859 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:53.851271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:53.873 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:41:53.865917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:03.977 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:03.968647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:03.991 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:03.981668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:14.084 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:14.076505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:14.097 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:14.089784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:24.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:24.184790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:24.206 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:24.198857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:34.307 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:34.299376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:34.327 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:34.319649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:44.409 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:44.400819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:44.421 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:44.413464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:54.480 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:54.472345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:54.525 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:42:54.517936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:04.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:04.624697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:04.645 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:04.637597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:14.720 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:14.712214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:14.732 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:14.724961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:24.831 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:24.823513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:24.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:24.838430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:34.945 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:34.937383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:34.959 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:34.951455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:45.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:45.015792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:45.035 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:45.028270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:55.133 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:55.125240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:55.145 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:43:55.137964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:05.222 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:05.213992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:05.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:05.228120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:15.339 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:15.331175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:15.349 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:15.342280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:25.434 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:25.426174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:25.447 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:25.439596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:35.566 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:35.557883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:35.578 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:35.570186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:45.681 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:45.673266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:45.693 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:45.686031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:55.798 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:55.790760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:55.810 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:44:55.802575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:03.197 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:03.189225 I | op-mon: parsing mon endpoints: c=10.111.142.38:6789,a=10.100.208.21:6789,b=10.110.240.83:6789 2022-03-09T02:45:03.859 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:03.851100 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:45:03.840+0000 7f89d42c8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:45:03.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:45:03.840+0000 7f89d42c8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:45:03.860 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09T02:45:03.840+0000 7f89d42c8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:45:03.860 INFO:tasks.rook.operator.smithi042.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:45:05.865 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:05.857648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:05.969 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:05.961226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:16.084 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:16.076785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:16.098 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:16.090889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:26.211 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:26.203280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:26.225 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:26.217729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:36.291 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:36.283892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:36.302 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:36.294680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:46.367 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:46.359383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:46.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:46.373651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:56.512 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:56.504992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:56.527 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:45:56.519622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:06.647 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:06.640106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:06.661 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:06.653610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:16.744 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:16.736160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:16.759 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:16.751677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:26.826 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:26.818446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:26.837 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:26.830480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:36.941 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:36.933520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:36.957 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:36.950005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:47.066 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:47.058774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:47.083 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:47.071881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:57.178 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:57.171025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:57.193 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:46:57.186251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:07.282 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:07.274873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:07.293 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:07.285758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:17.398 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:17.391363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:17.413 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:17.405707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:27.524 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:27.516956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:27.538 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:27.531151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:37.612 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:37.604521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:37.688 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:37.681386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:47.766 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:47.758893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:47.778 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:47.771061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:57.846 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:57.839422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:57.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:47:57.858652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:07.930 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:07.923258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:07.946 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:07.939063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:18.048 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:18.041307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:18.064 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:18.057282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:28.151 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:28.143468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:28.163 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:28.155789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:38.236 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:38.228940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:38.285 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:38.278066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:48.374 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:48.366675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:48.390 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:48.383303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:58.532 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:58.524591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:58.545 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:48:58.538785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:08.668 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:08.660716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:08.681 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:08.673725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:18.760 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:18.752989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:18.813 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:18.805297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:28.905 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:28.897899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:28.936 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:28.924379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:39.055 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:39.047933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:39.066 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:39.057534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:49.169 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:49.161963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:49.183 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:49.175995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:59.287 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:59.279881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:59.300 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:49:59.292508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:09.391 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:09.383847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:09.405 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:09.398226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:19.516 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:19.509339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:19.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:19.523694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:29.600 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:29.593052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:29.613 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:29.605943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:39.689 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:39.682117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:39.700 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:39.693639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:49.819 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:49.811768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:49.829 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:49.822093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:59.910 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:59.902829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:59.918 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:50:59.911495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:10.024 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:10.016844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:10.037 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:10.029830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:20.166 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:20.159103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:20.180 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:20.173123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:30.284 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:30.277416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:30.295 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:30.288545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:40.381 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:40.374460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:40.395 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:40.388201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:50.548 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:50.540375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:50.562 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:51:50.554643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:00.632 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:00.624892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:00.644 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:00.637076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:10.751 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:10.744269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:10.784 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:10.776638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:20.852 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:20.845276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:20.866 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:20.859696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:30.934 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:30.927437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:30.949 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:30.942093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:41.011 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:41.003749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:41.020 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:41.013807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:51.091 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:51.084046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:51.148 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:52:51.141853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:01.217 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:01.210446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:01.229 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:01.221930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:11.360 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:11.352352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:11.372 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:11.364794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:21.440 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:21.432180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:21.452 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:21.445600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:31.531 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:31.523864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:31.542 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:31.535855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:41.610 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:41.603488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:41.625 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:41.618642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:51.692 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:51.685677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:51.708 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:53:51.701360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:01.793 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:54:01.786161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:01.803 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:54:01.796185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:11.897 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:54:11.890504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:11.909 INFO:tasks.rook.operator.smithi042.stdout:2022-03-09 02:54:11.901929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T05:15:30.222 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:15:30.241 DEBUG:teuthology.task.console_log:Killing console logger for smithi042 2022-03-09T05:15:30.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi153 2022-03-09T05:15:30.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi178 2022-03-09T05:15:30.243 DEBUG:teuthology.exit:Finished running handlers